Peter Park

peterpark

I’m in the category ‘GraphQL’. Any vote would be appreciated! nominated for the Noonies 2022 The Firebase Realtime Database is a NoSQL database hosted in the cloud. Data is synchronized in real-time with every client on all platforms (Web, iOS, and Android). The data is stored as JSON and is shared by all of your customers when you create cross-platform apps. This ensures that all of your clients always have the most recent data available, even when offline. You can find the complete code for this article in our , or watch the video walkthrough on our . Github examples repository Youtube channel Firebase is a cloud-based platform from Google that allows you to build web and mobile applications quickly. It offers authentication, data storage, monitoring, logging, and deployment services. The Realtime Database is one of the services offered to developers. You can access the data from a Firebase Realtime Database using a REST API. But it lacks a GraphQL API, which we'll be creating using StepZen. Set up Firebase Realtime Database The first step to using a Firebase service is to create a free account in the Firebase console if you haven't already. You receive a brand-new Firebase app with a unique URL that ends in . This URL is used to store and sync data to the app's database and authenticate your users. firebaseio.com Within the Firebase console, you need to create a new project; let's name it "StepZen demo". The setup asks you several configuration questions, which you can ignore for the sake of this demo. Within a few minutes, your new Firebase project is ready, and you can continue by adding a Realtime Database by using the navigation on the left side of the page. When you press "Create database," you need to select the region you want the database to be created in. For example, the United States, Europe, and Asia. Also, you need to configure the security settings for the Realtime Database. At first, you need to enable "test mode" to make it easier to create and read our first data: By pressing "Enable" your database is created in the region that you selected. Depending on the region of your database the URL is either (for databases located in ) or for databases in all the other locations. YOUR_FIREBASE_PROJECT_ID.firebaseio.com us-central1 YOUR_FIREBASE_PROJECT_ID.REGION.firebasedatabase.app The final step is adding data to the database, which you can do from the Firebase console or through the REST API. To get initial data into the database, we'll be using the Firebase console to upload the JSON file that you can find in the Github repository for this post . By pressing the three dots on the right side of the screen, you can select "Import JSON". init.json here After uploading the data, you can see in the Firebase console that there are customers and orders in the database. That's it. Your Firebase Realtime database is ready to go. In the next step, we'll be creating the StepZen GraphQL schema. Generate a GraphQL schema You can retrieve data from the Firebase Realtime database using their REST API. An example curl request to get a single customer would be: curl 'https://YOUR_FIREBASE_PROJECT_ID.firebaseio.com/customers/-N6rOmxa7vOOTpZZSllL.json' With StepZen, we can use the same curl command to create a GraphQL API that retrieves the same data. First, you need to install the StepZen CLI using: npm i -g stepzen And run the command to convert the Firebase REST API to GraphQL. In this command, you can see the initial curl is used together with a few flags to name the generated query and response type: stepzen import curl 'https://YOUR_FIREBASE_PROJECT_ID.firebaseio.com/customers/-N6rOmxa7vOOTpZZSllL.json' --name customer --query-name getCustomerById --query-type Customer Note that if you haven't signed up for StepZen, your GraphQL API is deployed on a public endpoint. If you're planning to store sensitive data we recommend signing up so that an API key secures your GraphQL API endpoint. After the StepZen CLI has finished converting the REST API request to a GraphQL schema, several files have been created: myproject
  |-- customer
      |-- index.graphql
  |-- index.graphql
  |-- stepzen.config.json In the file , you can find the generated GraphQL schema. customer/index.graphql To deploy this GraphQL schema and query your new GraphQL API, you need to run: stepzen start The StepZen CLI then deploys the GraphQL schema and return your endpoint directly in the terminal. This looks like when you haven't signed up for StepZen. If you do have an account, two endpoints are available. https://public***.stepzen.net/api/with-firebase/__graphql a localhost endpoint that contains the GraphiQL IDE, and a production-ready endpoint that can be accessed by providing your StepZen API key. Learn more about the different endpoints in the . documentation Visiting this public URL (either or starting with ) lets you query the endpoint using GraphiQL: localhost https://public*** From here, you can query the GraphQL API and inspect what the response types in the schema look like. Let's continue in the next section by adding a query to get all customers at once. Transform API responses Besides getting a single customer, you can also get all customers from the Firebase Realtime Database. But it has one caveat, as you'll learn once we get the data using the REST API. The curl to get this list is similar to the request we've used in the previous section: curl 'https://YOUR_FIREBASE_PROJECT_ID.firebaseio.com/customers.json' And we can add it to the GraphQL schema in manually or by running another command. The second option has a caveat as the imported schema shows the message: customer/index.graphql stepzen import curl Result typed as JSON since it has fields ['-N6rOmxa7vOOTpZZSllL', ...] that are invalid graphql identifiers The request response to the REST API returns JSON in an unsupported format. The REST API returns a list of key-value pairs with field names representing the key instead of an array. Therefore you need to into an array to get a list of customers. If you've used the StepZen CLI to import the endpoint, you need to add the field to the query. Or, if you haven't, you can add the query from the code block below to the schema in : transform this object transforms getCustomers getCustomers customers/index.graphql type Query {
  getCustomerById(id: ID!): Customer
    @rest( 
      ##
    )
  getCustomers: [JSON]
    @rest(
      endpoint: "https://YOUR_FIREBASE_PROJECT_ID.firebaseio.com/customers.json"
      transforms: [{ pathpattern: [], editor: "objectToArray" }]
    )
} The response type for this query is now as the StepZen CLI couldn't generate the GraphQL type automatically as GraphQL doesn't support key-value pair lists. On the GraphiQL IDE for your endpoint, you can already query the results for , but you won't be able to select any fields. [JSON] getCustomers Therefore we need to add the GraphQL response type manually. Type was already generated when you imported the Firebase REST API endpoint to get a single customer and add a second type for the transformed key-value pair with the type : Customer CustomerList type Customer {
  email: String
  name: String
}

type CustomerList {
  name: ID
  value: Customer
}

type Query {
  getCustomerById(id: ID!): Customer
    @rest( 
      ##
    )
  getCustomers: [CustomerList]
    @rest(
      endpoint: "https://YOUR_FIREBASE_PROJECT_ID.firebaseio.com/customers.json"
      transforms: [{ pathpattern: [], editor: "objectToArray" }]
    )
} Note that you can also use the tool to convert the JSON response to GraphQL SDL automatically. JSON2SDL When you go back to the GraphiQL browser, you can now also query the list of customers and determine the fields that the GraphQL API should return: query {
  getCustomers {
    name
    value {
      email
      name
    }
  }
} Even though the response is the same as the previous query, you now have complete control over which fields are returned. If you'd run the same query and remove the field in , only the name of the customer will be returned. email value In the next section, we'll combine the customer information with the orders we've previously imported into the GraphQL schema. Combine data in GraphQL Next to customer data, we also have order data in the Firebase Realtime Database. To combine the customer data with the order data, we need to create the query to get the orders from the Firebase Realtime Database. Again you can use the StepZen CLI to convert the REST API to a GraphQL API. We'll start by importing a single order so that the return type is autogenerated: stepzen import curl 'https://YOUR_FIREBASE_PROJECT_ID.firebaseio.com/orders/-N6rOvLXK2t39x-Bp0UP.json' --name orders --query-name getOrderById This adds the new query with the response type to a new schema in : getOrderById Order order/index.graphql myproject
  |-- customer
      |-- index.graphql
  |-- order
      |-- index.graphql
  |-- index.graphql
  |-- stepzen.config.json Both the schemas are linked in in the project's root. index.graphql Using the query you can now retrieve a single order, for example: getOrderById {
  getOrderById(id: "-N6rOvLXK2t39x-Bp0UP") {
    carrier
    shippingCost
    customerId
  }
} You can see that this query also returns . This field references the customers we also have in our database. With the directive, you can use this value to retrieve the customer and link it to the response type for the order: customerId @materializer type Order {
  carrier: String
  createdAt: Date
  customerId: String
  customer: Customer
    @materializer(
      query: "getCustomerById"
      arguments: [{ name: "id", field: "customerId" }]
    )
  shippingCost: Int
  trackingId: String
} With this addition, you can retrieve the customer information for every order by adding the field and the subfields you want to include. customer Using you can combine any available data to query in your GraphQL schema. Not only data that is coming from the same database, but also data from external sources - such as another database, third-party REST APIs, and more. In the next section, we'll add logic to add more data to the Firebase Realtime Database. @materializer Use mutations to add and update data Finally, we'll add new data to the Firebase Realtime Database using GraphQL. The REST API from Firebase allows you to both . The request returns the ID of the updated or inserted value when successful. insert and update data Let's try to insert a new customer into the database by adding the mutation and response type below to your GraphQL schema in : customer/index.graphql type Response {
  name: ID
}

type Mutation {
  insertCustomer(name: String!, email: String!): Customer
    @rest(
      endpoint: "https://$project_id.firebaseio.com/customers.json"
      method: POST
      postbody: """
      {
        "name": "{{.Get "name"}}",
        "email": "{{.Get "email"}}"
      }
      """
      configuration: "firebase_config"
    )
} This mutation requires the query parameters and and passes their values to the underlying REST API using the field. The syntax is used here to get the value that's passed to the GrapQL Query. name string postbody {{.Get "VARIABLE_NAME"}} When you use the following query: mutation {
  insertCustomer(name: "Peter", email: "peter.parker@avengers.com") {
    name
  }
} The response looks something like this: {
  "data": {
    "insertCustomer": {
      "name": "-N7WtNDgHYjCrED--Bq3"
    }
  }
} You can validate if the customer with the name "Peter" has been added with the query and the returned ID. Also, you can find this new customer in the Firebase console: getCustomerById Not only can you insert new data, but you can also update existing data. You have two options when updating specific fields or replacing the entire JSON data for an ID. In this article, we'll use as will overwrite the data at the specified location, including any child nodes. PATCH PUT Adding the new mutation below to your schema in : customer/index.graphql type Mutation {
  insertCustomer(name: String!, email: String!): Response
    @rest(
      ## ...
    )
  updateCustomerName(id: ID!, name: String!): Customer
    @rest(
      endpoint: "https://YOUR_PROJECT_ID.firebaseio.com/customers/$id.json"
      method: PATCH
      postbody: """
      {
        "name": "{{.Get "name"}}"
      }
      """
      configuration: "firebase_config"
    )
} Allows you to update the name of the customer. Suppose you want to change the name of the customer we've just created from "Peter" to "Peter Parker", you can do it like this: mutation {
  updateCustomerName(id: "-N7WtNDgHYjCrED--Bq3", name: "Peter Parker") {
    name
  }
} Note that the response type of the query is instead of , as requests to the Firebase REST API return the updated value. updateCustomerName Customer Response PATCH That's it. You have now created a GraphQL schema for a Firebase Realtime Database. When you want to take this into production, don't forget to for your REST API endpoint from Firebase. enable authentication Conclusion In this article, we created a GraphQL API for a Firebase Realtime database using REST APIs. This GraphQL API lets you query and mutate data and make combinations between different entities. You can find the complete code in our , and watch the video on our . We would love to hear what project you start building with StepZen and Firebase. Join our to stay updated with our community. Github examples repository Youtube channel Discord I’m in the category ‘GraphQL’. Any vote would be appreciated! nominated for the Noonies 2022 You can find the complete code for this article in our , or watch the video walkthrough on our . Github examples repository Youtube channel Also published . here

Setup a GraphQL API for a Firebase Realtime Database With StepZen

Exploring a business model where startups are challenging the whole governments to heal the world faster than charities. What my Internet looks like Hi, my name is Pyotr. I’m from Russia. I use VPNs so you can easily reach me on LinkedIn and Reddit. You can also text me in Telegram because I’m using it’s proxies. I have a sophisticated system of passwords. I prefer pin-code for my phone over touch id. My disks are encrypted. I’m hiding all sensitive data with double bottom to protect it from thieves and (more importantly) from those who swore to protect me from thieves. Yes I’m a bit paranoid, but… People are getting . So I guess I do have a real danger of being arrested for the kind of shit I’m going to discuss here. Nothing will stop then an officer to “use” my finger (no, no, not separately from me, not that bad) to unlock my phone and get evidence against me. jailed just for likes and reposts With all these measures I’m still risking. But I value freedom and truth more. Telegram proxies I’m not unique Not at all. In Russia — and I believe in many other countries with authoritarian leadership — almost everyone who deals with Internet (meaning everyone) feels a bit constrained. The only way to feel free is to use some or all of the means described above. From one side it’s getting worse. Services get banned. Telegram, LinkedIn, archive.org, deviantart.com and many other seemingly random websites, like PornHub and the new design of Reddit (old.reddit.com is fine — try figure out the criteria!) are all unavailable in Russia. From the other side there is a technological response. Startups are already helping. There are literally billions of people who want to use services unavailable in their countries. Citizens armed with unstoppable technology are starting to oppose their own governments. I believe a huge market is hidden here, underestimated both by entrepreneurs (in terms of income) and governments (in terms of threat). In this post I would like to explore its opportunities from an entrepreneur’s point of view. Telegram example This year’s telegram story in Russia is a good example of Tech vs Gov fight. The fight which made Telegram way more advanced, brought an army of new users to it and made them super-loyal. It shifted Telegram in peoples minds into another dimension, now associating it with resistance, freedom and hope. On October 2017 FSB (Russian federal security service) to hand over users’ encryption keys. Company declined, payed $14,000 fine and found itself in a half-year-long futile debates. asked Telegram FSB is above Constitution. They don’t care for free speech and the right for anonymous communication. They just need the encryption keys. FSB is above logic. They don’t care end-to-end encryption means Telegram even doesn’t have the keys. They just need the encryption keys. Or they will shut down Telegram in Russia. On April 10th 2018 seeing argumentation is useless, Durov steps back and decides to send the “keys” in an “official” response to the FSB head: The joke originates from a . But as Telegram team , it became the only “official” response Russian Internet community is aware of. third party helped to promote it Less than a week after another Russian bureaucracy beast — media watchdog RosKomNadzor — started blocking Telegram. The fight had begun. RosKomNadzor blocking millions of IPs every day, Telegram creating even more. During the first couple of days over . The number of Telegram users in Russia is . Four million less than that! Another fact — 17 mln is 0,5% of all in the world! 17 mln ip-addresses were blocked 13 mln possible public IPv4 addresses One just cannot shut 0,5% of Internet down without consequences. First, a lot of “good” Russian websites hosted at Amazon AWS, Digital Ocean and other foreign hosting services were down. Second, Google, Viber, Facebook — all started to have blackouts but stayed more or less available. Then Asana (task manager), Dashlane (password manager), Trello, npmjs.com were down. This was the point when I started to use VPN just to continue work on my . After that came the total collapse. The Internet became useless. And not only Internet. People complained on . At an apogee RosKomNadzor charity blockchain project ATMs and credit card terminals not working even blocked itself! RosKomNadzor logo community revised 2 weeks after businesses were to lose $2 billions. RosKomNadzor finally listened to major Russian media companies, begging it to calm down. The Internet gradually became usable. I still can’t imagine myself without a VPN though. estimated From Durov’s official vk.com profile As it was often heard those days RosKomNadzor broke the Internet. But had it broke Telegram? Well, I couldn’t access Telegram for around 2 hours one day. Shame. Aftermath of this battle I believe by starting this battle Russian government made one of the most crucial mistakes in it’s modern history. They broke the Internet, damaged or destroyed business, wasted tax money. All for the battle with some phantom terrorists (but perceived as a battle for control over masses). Which is more important — they’ve lost this battle. Unsurprisingly it lead to protests and made the already growing antagonism of us vs them even stronger. and flew them into the air through out the country demanding Telegram blocking to stop (the protest was organized through Telegram). People made paper planes REUTERS/Tatyana Makeyeva. Not only in Russia Technology evolves so fast that no government is capable of keeping up with it. The smartest ones cooperate and those who try to fight technologies fail. Startups have more competence and will always by ahead. Durov understands this asymmetry. He had a competence to see the robustness of his app. He saw no positive outcomes of negotiations with the officials. He saw the opposition leadership vacuum in Russia. And he exploited the situation, mocking the officials with “keys”, organizing the paper planes campaign, promoting provocative artwork and urging people for “digital resistance”. Durov Digital Resistance logo. proposes A young smart outcast fighting the whole government. Can you imagine a better pr campaign? Having poor competence in technology Russian government had only succeeded in creating an important precedent. It showed that WE’ve got something THEY can’t fight with — the unstoppable technology. How can other entrepreneurs benefit? A feature that helped Telegram survive is it’s MTProto proxies. It allows anyone to create a telegram proxy and invite advertisers. Advertisers in turn can add their chat to user’s chat list. While using proxy a user cannot unsubscribe from this chat. This way users are paying with their attention to get freedom of communication. Telegram gets unlimited source of new proxies. Notice the Proxy Sponsor mark at the very first chat. The check-marked shield at the top means the proxy is active. A foreign company providing a service to resists the current regime. The regime can’t do anything about it. People are willing to pay for it. Providing basic human rights with unstoppable technology from an unreachable jurisdiction — sounds like a business model! For convenience of further conversation let’s coin a term for it. Freedom as a Service is too pessimistic as it suggests that you’ve already lost your freedom. Revolution as a Service is closer, but too romantic and suggests lost freedom too. What comes before you lose freedom and have to regain it through revolution? You resist. I’d suggest Resistance as a service (RaaS) is the best candidate. Let’s keep the definition simple and broad enough not to limit our further thinking. RaaS — a service providing access to basic human rights. Now that we have a term lets explore the business model further. We are still looking from an entrepreneur’s perspective. We need to understand opportunities and maybe generate ideas. Demand is growing and self-generated. Have a look at by country at Wikipedia and try to estimate the number of people in those countries. There are literally billions of people. The demand is already huge. Internet censorship and surveillance And demand is growing. Any government has to defend it’s sovereignty. Foreign influence is a threat. Defense lines are trenched through means of communication. But these are the same means people and businesses operate on. Push too little and you’ll probably lose power letting foreign ideas to leak in. Push too hard and you’ll find yourself slowing down economy and depressing people. Which may lead to losing power in a more dramatic way. some all A pragmatic solution would be to relax and stop fighting. Redirect huge fighting budgets into social programs. Doing good is the most cost effective strategy as one cannot influence a happy nation. But the most important goal of power — is to keep the power. Oppressive countries tend to deviate into fighting. The more they fight the more the demand to resist the more they fight the more the demand to resist… This is what we see in Russia — positive feedback loop. The demand for resistance is self-generated. More of unstoppable technologies are coming. RaaS is possible with existing VPNs, torrents, and a variety of peer-to-peer techs. But true unstoppable technologies are just about to come. IPFS Consider blockchains. With decentralization, trust, security and sometimes privacy out of the box blockchains got a crucial success driver — incentivization. People can actually make money by keeping the system secure. is the simplest example. Telegram with its MTProto proxies created a platform where advertisers met proxy providers. Blockchains will unleash all kinds of unseen interactions like that. Durov is already there by the way with his . Mining Telegram Open Network Credits: Project Loon Consider satellite or stratosphere Internet. It might be very close as , and are competing hard to deliver it. Today governments can only stop digital resistance by hurting themselves. In the nearest future there will be no chance at all. Unless somebody is considering shooting 650 satellites. Starlink Project Loon OneWeb Add up open hardware DIY devices and you have a fairly good primordial soup of technologies to play around. This mix of technologies is so powerful that frankly it scares me. I can imagine a world where governments are helpless (or even useless). No matter how liberal you are you don’t want a government to collapse. Let’s just gradually replace its functions. Competitors. There are no pure RaaS solutions. There is a which advertises uncensored Internet as its main feature. There is , which is a peer-to-peer networking tool and (“unblocking” twitter) powered by it. There is petitions platform . These may be considered pure RaaS solutions. VPN Lantern FireTweet change.org But usually censorship circumvention and RaaS come as a byproduct. Or a service qualifies as a RaaS only for a specific group of people. Depending on where you are and what you are doing. You can use Torrents or IPFS to download free music and videos, or you can share a book prohibited in your country. You can avoid ads with network, or you can build a censorship resistant opposition website. You can use VPN to stay safe on public WiFi or you can reach out to somebody on Facebook from China (or enjoy the new design of Reddit from Russia!). Tor The mere fact that there are so few of pure RaaS is an opportunity itself. Created with photofunia.com Ideas. RaaS is wider than just an Internet access. We have determined RaaS as a service providing access to basic human rights. This may sound a bit obscure, especially if you need no effort to receive these rights. Here are some example situations for you to figure out what human rights are ready to be outsourced (maybe a better definition of RaaS is outsourcing government functions?). Sorry for the long and pessimistic list: a huge bully keeps beating bar visitors and security is afraid to stop him because he is from police a women tries to get medicine under sanctions for her son and (happy end story, thank God) end up in prison as a drug dealer all the money you’ve earned the last month now worth less than a loaf of bread 40 children and no official is punished (damn it hurts). killed in a shopping mall fire you return from a war and find somebody occupying your house, there is no record proving it belongs to you you don’t trust elections (or you’ve got no choice) you’ve got no identity you’ve got no Internet access (no device, no line) These are still unsolved problems in many areas of the world. With technology and competence solutions may be simple and even profitable. This may sound cynical, but maybe it is the fastest way to make people happy. Maybe RaaS can solve problems which cannot be solved by charity and non-profits. Maybe it is worth trying to figure out a solution in terms of for-profit? Try asking new questions. How can we provide basic human rights as a service? Or — another way of thinking — how can we outsource government functions? Can RaaS be more efficient and even more ethical than charity? These questions may bring a lot of ideas. Demand is huge and growing. Resources are in abundance. Competition is little. And most importantly there is a great chance to make the world a better place. Resistance never dies If you are an entrepreneur in a safe place, have a cup of coffee and think about it. You probably got skills and resources to help a lot of people. to , that on and . Bad men need nothing more compass their ends than good men should look do nothing — John Stuart Mill If you are a citizen of an oppressive country. Remember after one hope is gone there is always another one. Resist.

Resistance as a Service

Or do you still think blockchains got no use? Technology reflects psycho It’s been hard times for almost all crypto this year. Blockchain technologies were almost buried. No need to select any particular one, it doesn’t matter much for this post. I use Ethereum as an example. Giving no preferences, I just know it best. So they say my beloved Ethereum is centralized, it’s dapps has no users and it’s attempt to deliver a world computer has failed. Sounds like blockchains are not perfect. Hm. But have we seen a tech like this before? Well, it takes time to grow from 0 to 1. They say blockchains got no use, I’d say we need to use our imagination better. In this post I would like to justify blockchains by giving yet another perspective on crypto industry. The perspective of world pains. Just like consumer products reflect our psycho (best described in ) I believe the same way our psycho determines technologies that we use. The world is changing and the Internet (as well as any other technology) is changing accordingly — reflecting our needs. Thus the success of a technology is determined by how well it is aligned with the way we think. The System of Objects by Jean Baudrillard In order to show that blockchain technology is aligned perfectly with the current world’s mindset I’d like to identify some of the and show how blockchains can help. most acute world’s pains While there are many amazing blockchain solutions. There is Golem for decentralized computation. There is Maker DAO which despite all criticism is still working after ETH-USD pair dropped so low. There is FOAM — mind-blowing decentralized geo-positioning system. And there is Augur — the world’s first assassination platform… Of course not — it is a part of Ethereum criticism too. Actually Augur is the first of its kind prediction market and it is amazing. These projects are completely novel and with far-reaching benefits, but they are hard to understand. It is hard to predict their future and almost impossible to draw parallels with something which already exists, thus making it harder to see the significance of blockchain technology. Let’s compare something simpler You’ve probably heard of a guy (Alex) who earned 1mln dollars for his college by selling pixels . The idea was to have a web page with 1000x1000 pixels field where anybody could buy 1 pixel for 1 dollar and advertise. Alex succeeded, sold all his pixels and the project did became a part of the Internet history as was advertised. back in 2005 Almost the same project is now live on Ethereum. It is called (MEH). It has the same 1000x1000 pixels filed. But unlike the original it also has rent functionality and ability to sell pixels through a third party exchange. The price starts with $0.01 per pixel, and doubles every 10% of pixels sold. The Million Ether Homepage These two projects are very similar and easy to compare. Let’s imagine now that the original project has never existed. And Peter (author of MEH) is doing just the same thing as Alex did 13 years ago. We’ll see how hard it is to succeed with the old tech now and how blockchain can help. Next I will identify 3 of the most acute world’s pains. For each pain I’ll show hopes which are now on the horizon thanks to blockchains. And then compare business solutions with respect to their ability to address the pains. Pain: Trust The first pain is trust. All of us are products now. Our data and our attention are more valuable than ever. And there are so many new ways to get attention and to collect our data. And there will be more. . I see no reason for this tendency to change. 90% of all data was created in the last 2 years With data becoming so valuable it’s not enough to just trust some company and it’s software to treat it right. We want to be 100% sure what our programs do. We are deleting Facebook and just because of that. leaving Google Chrome Trust is our pain. And we need software we can trust. Hope: Blockchains give us hope for just that. Ethereum’s core feature is deterministic (read trusted) execution of smart contracts. The code of smart contract is open, you can see exactly what it is doing and you know that no other code will be run. Maybe it sounds humble, but it is a huge thing. Huge because now businesses are able to develop solutions meeting our current demands — software we can trust. Solution: Looking at our pixel map project. It’s main functionality is buying pixels. What would you prefer: Pay with your credit card to some random Russian h̶a̶c̶k̶e̶r̶ developer promising you to deliver pixels with responsibly “insured” with a 21 page user agreement valid in an unreachable jurisdiction. …or… Look at the publicly available smart contract code. See what it’s functions exactly do. Try to run it on testnet, try to break it. Make sure the contract is secure. Only then send money to the main net contract and get your pixels in the same transaction. Ok, the second option is a bit from a perfect world. Most users will skip the code the same way they skip every user agreement they meet. But it illustrates very well fundamental technological advantage — no matter if you do your research or not — there is at least a to see how exactly your money move. possibility For a small project — this is a line between success and failure now. Making a project like MEH closed sourced nowadays wouldn’t satisfy the world’s need for trust. It is Ethereum that makes the project up-to-date. Scale up: MEH is a very small step. But it shows that businesses are now able to offer solutions which can meet the demand for trust. Scaling up imagine you could control the amount of data you allow Facebook and Google to access to (guaranteed by code, not just worlds). Or imagine a research lab getting access to somebody’s medical data without revealing their names. The demand for trust is huge and solutions will continue to emerge. Pain: Trust Yes, trust again. And data again. But not your data, this time — their data. Just like our software, most organizations or entities we deal with (banks, governments, corporations, media, etc.) are black boxes too. Do exchanges manipulate markets? Do governments forge election results? Is this real news or Fake news™? We cannot be sure. But we want to be sure. Trust is our pain. And to the same extent as we want trusted software we want trusted organizations. Hope: While we cannot put every organization on code (yet) we can make them transparent. Blockchains now let us build systems with 100% transparency and history stored forever. If trust cannot be built in with code, it can be built over time through perfect transparency. The more history records we see, the more trustworthy an organization or project becomes. Solution: Going back to our project. 13 years ago this genius guy Alex earned money for college. Today Peter, not so genius maybe, but probably more altruistic is giving away 80% of his income to charities (I can call Peter whatever I want because Peter is me). What would you prefer: a tweet on spending $800k on charity or a complete record of all charity transactions kept by a trusted third party? In MEH 80% of income is automatically stored in a special charity vault inside smart contract. These funds are then forwarded to a list of charities accepting Ether. Maximum income of the project is $1mln. If the goal is reached $800k will be donated to charities. All transactions can be tracked on chain (plus a for convenience). Google spreadsheet file Sure with 13-years old technology charity transfers could be confirmed too. Charities could announce publicly they’d received funds. But now it is much easier, as transparency is available out-of-the-box, as a by-product of how Ethereum works making it universal source of truth. It is now possible to make tiny donations and keep publicly available record of them. Scale up: It is possible to make organizations transparent without blockchains. It’s just those who owns data were and are not interested. With Ethereum bureaucracy can be moved on chain very easily. The pressure is higher now. There are less arguments no to do it. I personally see no serious arguments not to put real estate registry on chain for instance. Pain: Freedom Next stop — digital freedom. Decentralization is what societies benefit the most from throughout the history ( by Jared Diamond). Decentralization means competition, competition means progress. More importantly decentralization means choice and choice means freedom. Guns, Germs, and Steel Google’s ex-CEO Eric Schmidt says the internet . To me it sounds a bit optimistic. As China, Syria, Iran, Vietnam, Egypt and many more are already splitting. In Russia Telegram, LinkedIn, PornHub, and the new design of Reddit are not available (try figure out the criteria). In March 2014 When Turkish government blocked Twitter, VPN provider Hotspot Shield . will split in two by 2028 acquired 2 mln new users in just 48 hours …because above all we need freedom. Hope: Hope for freedom is what sells Blockchains best. Thanks to blockchains it is the first time in history we can build world-wide decentralized systems with economic incentives strong enough to keep these systems alive for a very long time (forever maybe?). Blockchains provide availability, sustainability and choice out-of-the-box. Businesses are now able to build solutions which inherit these properties by default. Meeting the demand for freedom. Soultion: MEH for example gives you freedom to chose the way you want to interact with it: You can chose which parts of the project you want to use. The web page is just a representation of what is being executed on the blockchain. You can copy it and make your own mirror (a 3d-version, an animated history version, nsfw-free or nsfw-only versions). Each 10x10 pixels block is a standard token ( ). It means that you can chose whether to buy and sell pixels on the platform or allocate it on a third party exchange. Or you can create an alternative rent marketplace, for example. ERC721 You can even copy the whole project and create your own ad platform. There is no way to create these properties using 13 years old technology. Scale up: Freedom of selling pixels the way you want may not sound so groundbreaking and may not be a huge step towards your personal freedom. But it is a step. And the road is long and fruitful. Imagine you could “unplug” yourself from one government and plug into another. There are many articles provisioning the . But got too excited. Let’s imagine some down-to-earth examples… whole governments on Ethereum Move your favorites from one music streaming service to another, use your credit card history to get discount at airbnb without revealing it, migrate your RPG-character from one game to another keeping experience points and converting inventory items to its equivalents. There is a huge room of innovation. All thanks to the fact that blockchains allow businesses to meet the demand of freedom. Wrapping up They say there are hacks, centralization issues, bugs. But please, it is so very early days. 2–3 years is an infant age for almost anything. Sure there is a heap of problems. But technology is here, and businesses are now starting to compete. Give them time to learn how to use the technology right. In the beginning I accused journalists of lack of imagination. I do lack imagination too. After all I’m trying to make money by copying an existing project. I cannot imagine all possible solutions blockchains are enabling. And I do see problems too. But I’m sure the problems will be solved and solutions are doomed to emerge. Because the world is waiting for it and demand is high. …If you liked the article, please consider buying some of my . best and freshest pixels

Trust, trust, and freedom: how blockchains can cure the world’s most acute pains

My setup to push smart contract data to website — scalable, secure and almost free The problem — how to connect my smart contract and website One of the key features of and its underlying smart contract (a decentralized advertising platform) is that all income is shared between referrals and charity. It means that there is a scenario where I get very little or no money at all (as I’m just a referral too). This sets the first requirement to my infrastructure connecting smart contract and the Internet — it has to be very low cost or free. The Million Ether Homepage The second requirement comes from my main goal. Which is to get as much hype as possible in the press. Which in turn means that the platform has to endure very high and inconsistent load. So cheap shouldn’t mean compromised speed or security. And of course the platform has to be user-friendly and reasonably simple to develop and maintain. In other words I need a good balance between cheap, scalable, secure, simple and user-friendly. The solution I decided I’m not gonna attach any custom GUI to my smart contract (at least for now, at least for ). As the threshold of getting into blockchain is already quite high I thought it won’t be much higher if I offer my users a GUI auto-generated by blockchain client (see this to see how GUIs are attached). This will let me build super-lightweight and super-simple web page. The web page will just represent the state of the smart contract, requiring very little traffic. MVP entrepreneur’s guide on smart contracts is a 1000 by 1000 pixels field where anybody can buy can buy (or sell) pixels and place ads. Meaning I need to store ads images somewhere. I decided to use Imgur.com for this to minimize traffic even further. The Million Ether Homepage Any smart contract by it’s definition is already cheap and scalable (and I worked hard to make a secure, simple and user-friendly one). In turn Imgur is an infinite everlasting source (free, scalable, secure, etc.). So the real question now is how to make my smart contract interact with the outer world. Or . how to push data from Ethereum blockchain to website My solution consists of 5 parts: provides UI (with no G) in addition to business logic and database Smart contract hosts images Imgur.com at droplet provides blockchain access Geth Ethereum node Digital Ocean Python app (polyether.py) at the same DO droplet listens to the contract, processes data and pushes it to website TheMillionEtherHomepage.com hosts the website Google App Engine Pushing smart contract data to www Next I will show you what happens when a user places new advertising. Or in other words uses function of the MillionEther smart contract. placeImage Notes in italics like this show my decision making, additional info and alternatives. How it works 1. User uploads an image to Imgur function of the contract requires a link to an image. So the first step for a user is to get it — drag&drop an ad to imgur.com and copy the link. placeImage I stick with Imgur.com, because everybody knows it. It’s free for non-commercial and open-source projects. It has huuuge rate limits “approximately 1,250 uploads per day or approximately 12,500 requests per day”. It has API and python library . All I had to do is register my application , get a token (application password) and start using it. Alternatively to Imgur I could offer users to encode their images in base 64 and send those bytes directly to the smart contract. But this way the contract will not be able to work with any image hosting or swarm storage (a sentence further). Meaning being too inflexible and too complicated for a user. Another solution is to use Ethereum’s swarm storage . But it is still too beta and too complicated for me. I’m quite sure though my contract is compatible with the swarm and I’ll be able to switch to it later. For now — Imgur. 2. User interacts with the smart contract User interacts with the smart contract through a GUI auto-generated by a blockchain client ( or ). If you are not familiar with the process check out this . MyEtherWallet.com EthereumWallet desktop app video Placing an ad requires sending desired coordinates (within 1000x1000px field), image url (copied from imgur), link to a user’s website and an advertisement text to the function. Once user submits the transaction the contract assigns an ID to the image and fires an event: placeImage NewImage(numImages, fromX, fromY, toX, toY, imageSourceUrl, adUrl, adText); After the transaction is mined into a block it’s receipt with event logs becomes available to any synchronized blockchain client. This is how looks like at Etherescan.io blockchain explorer (click to see something human readable). placeImage function receipt Convert to Ascii There are two ways of getting smart contract information: constant functions (those which don’t modify contract state, i.e. write to contract) and events. The easiest way to access contract data from an external app is to use events. The only way for another contract to access contract data is to use function calls . My polyether.py listens to events but the MillionEther contract has constant functions too to let other developers attach their own extensions (contracts). 3. Geth downloads a block with the transaction At Digital Ocean droplet Geth is run with this command: geth --cache=16 --rpc --rpcport 8545 --rpccorsdomain "localhost" Letting any local app to interact with it through . JavaScript API Geth is constantly downloading new blocks (syncing). And now it receives a block with the event. I use the cheapest Digital Ocean droplet with attached 40 GB volume. It lets me run full Ethereum node and my python app simultaneously. Here is how it’s configured . There are several ways to connect to geth : ipc (through a local file), rpc (through http protocol) and ws (through web-socket). I use rpc. Will show its benefits below. 4. Python app asks geth if there are any new events and processes them if there are any Polyether.py runs at the same DO droplet as Geth. Every minute it asks Geth if there are any new events. With EthJsonRpc library listening to evens is a breeze. Here is pretty all of the code needed to get all logs of a particular smart contract event (I’ll publish my ethereum_gateway.py soon — follow me to catch the post). self.client = EthJsonRpc('127.0.0.1', 8545) params = {       "fromBlock": hex(from_block),       "toBlock": "latest",       "address": 0x15dbdB25f870f21eaf9105e68e249E0426DaE916, # million_ether contract address       "topics": [event_definition["signature"]]      } logs = self.client.eth_getLogs(params) Both Geth and polyether.py are monitored by monit to prevent them from crashing (monit auto-restarts both if something bad happens). If a set of blocks has any new events of the smart contract polyether.py receives and decodes them. The app then verifies Imgur links, downloads and stores new images, generates a 1000 by 1000 pixels image and uploads it back to Imgur (the url is received in return). NewImage MillionEther With Infura.io (remote blockchain client) and pure JavaScript I could generate 1000 by 1000 image right in the user browser. But the image is rendered locally for two reasons: 1. The query to the blockchain node to get all ads images is very heavy and it would be getting heavier with time (as the number of blocks to check for events grows fast). It is cheaper to have all past events and images written to local database and to search for new events within only few blocks every other minute. 2. Generating 1000 by 1000 image out of 10000 small ones is unbearable task for the majority of computers. Using the link to the 1000x1000 image and ads details polyether.py generates new web page (the whole web page html), hash-sums it and sends it in JSON format through POST method to a secret GAE url with a long password. I could pass raw data to app engine and generate the web page there, but I’m already paying for the DO droplet. So I’m getting most of it to minimize app engine load. 5. Google App engine receives the web page The GAE app receives the POST request, checks the hash sum, the password and puts the web page into database. As far as I understand I could use App Engine API instead of those POST requests and write data to the database directly, securely, reliably and out of the box. But Google documentation… well, you know… it’s complicated. There are 6 ways (at least) you can store data at Google Cloud Platform: Bigtable, Cloud SQL, Datastore, Storage, Persistence disk and Spanner. I’m using Datastore — “a highly-scalable NoSQL database for non-relational data”. I find it the simplest. In addition every DB query is automatically cached. My web page though stored in database is usually loaded from cache, which has much higher free quotas and is much cheaper beyond them too. Alternatively I could store my web page at Storage, which is (presumingly) designed for static files and which is (probably) cheaper but… again… Google docs. 6. GAE delivers web page to a user Typically a user arrives through a url like this . The 42 character string after slash is an Ethereum address of a referral which invited the user. http://themillionetherhomepage.com/0xF51f08910eC370DB5977Cff3D01dF4DfB06BfBe1 Referrals addresses get to the GAE app the same way as images do: function of the smart contract fires event -> Geth -> polyether.py -> GAE and the address is saved in the same database. signIn NewUser The only load then app engine undergoes when somebody visits the link is to check if the address is in the database and load the current version of the web page. Both of these operations are usually performed using cached data, which is fast and cheap. I use Google engine for its scalability and for its free quotas . I think my super-lightweight app should be able to serve around 1000 requests a day for free and for little money above that. Conclusion I wanted cheap, scalable, secure and simple infrastructure to push data from Ethereum to the Internet and keep the platform as user-friendly as possible. Here is what I got. I to the blockchain and it will stay there for free forever. I pay $5 a month for DO and $4 a month for additional disk space (which I’ll soon get rid of). Still $0 for app engine. Imgur is free. Docs are for free at ReadTheDocs.com. Video instructions for free on Youtube. Cheap. paid 20 cents to deploy my smart contract The smart contract may experience literally any load (as much as the whole Ethereum blockchain may handle). Imgur is bottomless. Digital ocean setup has no heavy tasks and can be duplicated on any computer if needed (haven’t I already said that?). App engine if paid well can bear more load than the whole Imgur.com. Hopefully cached data will save my bills. Scalable. Assuming my smart contract is secure (and I worked hard to follow these ) the only point subject to attacks is the website. But there is almost no code there (and nothing to steal). Plus I’m quite sure (as I’ve seen it somewhere in the docs, but cannot find it any more) my app engine has DOS protection out of the box. Secure. official safety rules Simple. I spent a great deal of time to build all this up. I can’t say it was simple, but it surely was fun. And I hope I made it simpler for you. User-friendly. As user-friendly as blockchain technology currently lets it to be. Did I meet my requirements? I think I did. Of course there is no limit to perfection… Bonus. How to make it work even better (and cheaper) Currently as I’m running full geth node I’m cluttering with chain data 7 GB of disk space every month and have to buy more and more GBs. I can’t make up with it. I want to get rid off the local node and switch to remote node by Infura Buying additional disk space At it’s core it is as simple as changing only one line of code: - self.client = EthJsonRpc('127.0.0.1', 8545) + self.client = EthJsonRpc('mainnet.infura.io/my-secret-token',443,True) # True for ssl Just change http to https and localhost to infura.io and it works! Beautiful right? And even more beautiful if backed up by another node (e.g. ) and a local . Etherscan light client If you find this useful please like, share and follow me to get more articles like this in the future. See you! Thank you for reading.

From the blockchain of Ethereum through the Digital ocean and to the Google clouds

An entrepreneur, programmer and user walk into a smart contract — The ultimate Ethereum blockchain…

Monit monitors Geth, autorestarts it on crashes and sends email alerts. Good, good Monit! I’m running geth and back-end on a low-cost Digital Ocean (DO) Ubuntu droplet. I use to autorestart geth and send email alerts when something goes wrong with the processes. TheMillionEtherHomepage.com monit Here I’ll cover a working solution and a beautiful one which isn’t working (hope somebody will eventually fix it). Install and setup monit $ sudo apt-get update $ sudo apt-get install monit Backup the config file and start editing it with nano: $ sudo cp /etc/monit/monitrc /etc/monit/monitrc-orig # backup $ sudo nano /etc/monit/monitrc Allow access to monit status. Uncomment these lines: set daemon 120 # number of seconds between checkups set httpd port 2812 and     use address localhost # only accept connection from localhost    allow localhost # allow localhost to connect to the server Let monit include *.cfg files from its conf.d directory. Uncomment (or add at the very end): include /etc/monit/conf.d/*.cfg Set up mail server to send alerts. Add these lines if you are using gmail: # GMAIL set mailserver smtp.gmail.com port 587     username "username@gmail.com" password "12345678"     using tlsv1     with timeout 30 seconds 

set alert username@gmail.com # monit will send alerts to this address Exit nano editor: Ctrl+X to save, type Y, press enter, and enter again when it asks for a filename (leave the same filename). Set up permossions as monitrc now has very sensitive data. $ sudo chmod 600 /etc/monit/monitrc For better understanding of monit: Video — Monitoring Linux Systems and Services with Monit Digital Ocean: How To Install and Configure Monit Setup geth monitoring — the working solution Create and start editing geth.cfg file with nano: $ sudo nano /etc/monit/monitrc/geth.cfg Paste this code: # GETH CHECK PROCESS geth MATCHING "[g]eth.*fast"     start program = "/bin/bash -c 'geth --fast --rpc --rpcport 8545 --rpccorsdomain localhost --cache=16 >/dev/null 2> /home/my-logs-folder/geth.log'" as uid username as gid username     stop program = "/bin/bash -c 'kill -HUP $(ps aux | grep '[g]eth' | awk '{print $2}')'" as uid username as gid username     if not exist then alert     if not exist then restart Ctrl+X to save, type Y, press enter, and hit enter again when it asks for a filename. We use >/dev/null here because geth outputs everything to the stderr (standard error output). And we save all stderr output (i.e. all logs) to the /home/my-logs-folder/geth.log file. We use -HUP signal because -SIGINT (which you typically send with Ctrl+C) will not reach geth here. Restart monit: $ sudo monit reload This config will perfectly send an email to you when geth crashes and will perfectly auto-restart geth But. The disadvantage here is that it cannot send “graceful” -SIGINT signal to geth through monit. And have to send -HUP instead, which is presumably not safe ( ). I tried it and it works, but I prefer to send -SIGINT manually instead: Is it safe to kill geth with SIGTERM? $ sudo monit unmonitor geth # so that monit won't restart geth again $ PID_OF_GETH=`pidof geth` # get geth's process ID $ kill -2 $PID_OF_GETH # KILL IT NOW!... but do it gracefully. This is clumsy. I’ve tried to make it more graceful but failed. Posted this question at stackexchange — How to monitor and auto-restart geth with monit? Setup geth monitoring — the right way I know that the config should look very similar to this. geth.cfg # GETH CHECK PROCESS geth with pidfile "/home/username/monit/geth.pid"     start program = "/bin/bash -c '/home/username/monit/geth.sh start'" as uid username as gid username     stop program = "/bin/bash -c '/home/username/monit/geth.sh stop'" as uid username as gid username     if not exist then alert     if not exist then restart geth.sh #!/bin/bash PID='/home/username/monit/geth.pid' LOGS='/home/username/monit/geth.log' KEYS='\--fast --rpc --rpcport 8545 --rpccorsdomain localhost --ipcpath /home/username/.ethereum/geth.ipc --cache=16' case $1 in     start)         setsid geth "$KEYS" &> "$LOGS" &         echo $! > "$PID"     ;;     stop)         kill -SIGINT `cat "$PID"`         rm "$PID"     ;;     *)     echo "usage: geth.sh {start|stop}"     ;; esac exit 0 Beautiful isn’t it? If you know how to make it work please submit your answer here — How to monitor and auto-restart geth with monit? Get monit status After you restart monit, you can start geth with the following command: $ sudo monit start geth This will run geth, autorestart it on crashes and notify you via email if something goes wrong. You can also check the monit status by the following command: $ sudo monit status And have a screen like this (mlneth here is python back-end of my ): project Monitoring geth Happy monitoting and no alerts to you all!

How to monitor geth and autorestart it on crashes with monit

Run geth with just 512Mb of RAM This is how I run full geth node alongside python back-end on a low-cost Digital Ocean (DO) Ubuntu droplet. With this config I’m in full sync (currently at block 3097083) with no crashes at all. TheMillionEtherHomepage.com Create a droplet Buy a 512 Mb RAM droplet and attach at least 20 GB volume to it (DO’s block storage). Just click button and DO will guide you through. Create droplet Setup a swapfile I tried to run geth with no swap, but it crashes in about 10–30 minutes after start. Here I’m setting up 2 Gigs of swap. One Gig is not enough. I had one kill with it within 2 mln blocks. $ sudo fallocate -l 2G /swapfile # Creates a file of a preallocated size $ sudo chmod 600 /swapfile # adjust the permissions $ sudo mkswap /swapfile # set up the swap space $ sudo echo '/swapfile none swap sw 0 0' >> /etc/fstab # enable swap $ sudo echo 'vm.swappiness=30' >> /etc/sysctl.conf # optional. The default swappiness (i.e. system willingness to use swap) is 60. You can set it to 30 if you care. As swap gradually kills SDDs. In my experience 30 is enough and it feels more eco-friendly. $ sudo /sbin/shutdown -r now # reboot $ free -mh # now you should see your swap usage. Digital Ocean’s instructions on swap are available here: How To Add Swap on Ubuntu 14.04 Install geth $ sudo apt-get install software-properties-common $ sudo add-apt-repository -y ppa:ethereum/ethereum $ sudo apt-get update $ sudo apt-get install geth Official Installation Instructions for Ubuntu Put geth’s database to the block storage volume You can run geth with key ( ), but I find the following approach simpler as you may have other software using the same ~/.ethereum folder by default. So we gonna create a symbolic link. datadir “/mnt/your-volume-name/ethereum/” Geth Command Line Options $ sudo mkdir /mnt/your-volume-name/ethereum # create an empty directory $ sudo chown username:username /mnt/your-volume-name/ethereum/ # set directory owner to username (you) $ sudo chmod 775 /mnt/your-volume-name/ethereum/ # set directory permissions $ ln -s /mnt/your-volume-name/ethereum/ ~/.ethereum # create symbolic link from your home directory to the volume Run geth and enjoy! $ geth --fast --cache=16 $ ls /mnt/your-volume-name/ethereum/ # see if geth is writing to the rigth place And add stats to enjoy even more. Add a cron task through: $ crontab -e At the bottom add the following task. It will write your attached volume usage to the disk.log file every day at 4 A.M. 0 4 * * * echo `date` `df -m | grep sda` >> /home/username/disk.log And in a couple of days check the logs to see your disk usage stats. If you see disk space running low, it is time to think of buying up more space. $ tail /home/username/disk.log Sun Jan 22 04:00:01 UTC 2017 /dev/sda 20031 14582 4410 77% /mnt/volume Mon Jan 23 04:00:01 UTC 2017 /dev/sda 20031 14809 4182 78% /mnt/volume Tue Jan 24 04:00:01 UTC 2017 /dev/sda 20031 15041 3950 80% /mnt/volume Wed Jan 25 04:00:01 UTC 2017 /dev/sda 20031 15294 3698 81% /mnt/volume Thu Jan 26 04:00:01 UTC 2017 /dev/sda 20031 15522 3470 82% /mnt/volume Fri Jan 27 04:00:02 UTC 2017 /dev/sda 20031 15739 3253 83% /mnt/volume Sat Jan 28 04:00:01 UTC 2017 /dev/sda 20031 15956 3036 85% /mnt/volume Sun Jan 29 04:00:01 UTC 2017 /dev/sda 20031 16161 2830 86% /mnt/volume Mon Jan 30 04:00:01 UTC 2017 /dev/sda 20031 16041 2950 85% /mnt/volume Tue Jan 31 04:00:01 UTC 2017 /dev/sda 20031 16271 2721 86% /mnt/volume Set up monit to monitor Geth and enjoy even when you go to sleep. To feel safer you can . It will restart geth if anything happens to it and inform you by email. set up monit to monitor Geth status Comments, questions, improvements? You can post them here or at the stackexchange thread: How to run geth at 512Mb RAM Digital Ocean droplet? Had two crashes: on 17th and 25th of March. So I highly recommend or other software. UPDT: monitoring geth with monit

How to run geth at 512Mb RAM Digital Ocean droplet

Reviewing The 2020 Tech Lead Developer Survey by Couchbase (Podcast Transcript)

We’re living in a world where machines are learning to tell the difference between dogs and wolves. Now, a robot knows not to pet a wolf! We’ll get back to this stuff in a second. For now, let’s introduce Evgeniy, a senior data scientist at . In our interview, we talk about a lot of awesome things (including dogs and wolves!). Let’s begin. VEON You have a Ph.D from Lomonosov Moscow State University in mathematical modeling. What led you into the field of statistics? Yes, my thesis dealt with non-negative matrix factorization. I guess my interest in statistics stems from starting programming when I was young. I’ve always liked to analyze and create. But my journey into statistics truly all started as a joke with my friend. We both knew statistics to be this super complicated subject, so we applied just to see what would happen. We both were accepted and just decided to do it. It turns out we loved the field. It’s funny how it all happened. I just really enjoying working with statistical models, especially when it involves real data. After school, when I analyzed data from medical and biological research, I realized this was the direction for me. A simple joke ended up being my lifelong pursuit! We discussed learning, research, and analysis a lot. Now that the age of AI has begun, what insights do you have that can help navigate this new world successfully? It’s an exciting world. Things like . Neural networks are designed to learn like the human brain, but we have to be careful. neural networks are everywhere This is not because I’m scared of machines taking over the planet. Rather, we must make sure machines learn correctly. _Abstract: Despite widespread adoption, machine learning models remain mostly black boxes. Understanding the reasons…_arxiv.org [1602.04938] “Why Should I Trust You?”: Explaining the Predictions of Any Classifier It didn’t learn the differences between dogs and wolves, but instead learned that wolves were on snow in their picture and dogs were on grass. It learned to differentiate the two animals by looking at snow and grass. Obviously, the network learned incorrectly. What if the dog was on snow and the wolf was on grass? Then, it would be wrong. One example that always pops into my head is how one neural network learned to differentiate between dogs and wolves. So you believe there must be great attention given to the way these machines learn, right? Absolutely. It’s tremendously vital. Not that I’m saying being able to differentiate between dogs and wolves isn’t important, but neural networks have the power to do incredibly important things. _Cloudy with a chance of tanks_medium.com Picasso: A free open-source visualizer for CNNs There is an example of when . The problem was that photos of the tanks were taken on cloudy days, while photos of the forest were taken on sunny days. The neural network passed all the tests successfully, but it was merely distinguishing between clouds and sun — not tanks and forest. It learned the incorrect way to differentiate the two. an army used neural networks to distinguish between camouflaged tanks and plain forest By sequentially blocking out parts of the image, we can tell which regions are more important to classification. This image was classified by the model, with a 94% classification probability of “tank.” Bright parts of the image correspond to higher probability of the given classification. For instance, And conversely, the tank tread regions are darker because without them, it’s hard to for the model to know if it’s looking at a tank. VGG16 the sky regions are very bright because occluding the sky doesn’t affect the probability of this image being classified as a tank. In both these examples models erroneously tuned to the implicit bias that was in the samples — that actually happens quite often (here’s ). That’s why it is important that we understand how a model makes its decision. another example AI can save lives, protect the environment, and overall help us build a better future. Clearly, this technology can get complicated and we not only have to make sure what we’re doing with it holds value, we must also handle it cautiously. Discover all our Data Science and Engineering positions HERE Meet Evgeniy! Can you tell us more about your career path after university? Now, I’m one of the , but this job role, data scientist, wasn’t really a common term when I started. You could call me a statistician as well. senior data scientists at VEON “All data scientists follow a crazy career path. It’s like a cool roller-coaster through all sorts of new territory — like artificial intelligence (AI).” Over my career, I’ve worked on medical research data for research institutions. I’ve done budget optimizations and time series forecasting for pharmaceutical companies and supermarket chains. Before VEON, I worked in Moscow for , the largest search engine in Russia. I specifically worked at the Yandex Data Factory, helping external clients leverage data for more success. I guess Yandex was where I technically became a data scientist. Yandex You have a breadth of experience. What have you learned along the way? My jobs after university have been about learning and reaching solutions by collecting and analyzing numbers and information — that’s the main point. The biggest thing I’ve realized is the power of data. For instance, in medicine, understanding how to utilize findings in correlation and causation can save lives. Also, I should mention I’ve become very interested in , which are booming in use in statistics right now. As we get better at discovering causal relationships, data will only get more powerful. causal graphs Maybe soon being a data scientist will be just as cool as being a techno star. Did I mention that I like techno? Let’s have a listen: What made you join VEON? that’s undergoing an incredible digital transformation. The culture here allows for lots of creativity. There are two other main reasons. VEON is an exciting company One, VEON is sitting on a goldmine of data. I’m having a blast going through all of it. There’s a lot of rich stuff and I’m learning so much. There are just so many ways we can use this data to enrich the lives of our customers. Evgeniy’s cat enjoying the garden Two, the location in really attracted me. It’s beautiful here. I love watching the birds chirp in the trees, my cat enjoying the garden, and the . The city also has a great , which is important for me. Amsterdam cherry blossoms in Westerpark work-life balance cherry blossoms in Westerpark We heard you teach a course. Can you share with us some details? I actually have with my ex colleagues. It is about machine learning and data analysis, and I specifically teach applied statistics. a specialization we made for Coursera I’m really proud of the courses because we go beyond the basic stuff so students can actually use what they’ve learned in real-world situations. For instance, we explore advanced things like and play with real data-sets. multiple hypothesis testing What do you do for inspiration outside of work? For personal inspiration, I read a lot about statistics and data science. One book I like is , which gives you a great picture of where we’re coming from and where we’re going in the data world. Computer Age Statistical Inference _The twenty-first century has seen a breathtaking expansion of statistical methodology, both in scope and in influence.…_web.stanford.edu Computer Age Statistical Inference: Algorithms, Evidence and Data Science I also read . Hyndman has written the best and his advice and opinions on forecasting are very valuable. Rob J. Hyndman’s blog R package for time series forecasting Another blog I like is one written by , an econometrician most famous for the . Sometimes he writes about mind-blowing results that went rather unnoticed (such as, for example, the proof that conditional mean minimizes any Bregman divergence, including asymmetric ones — it is just nuts). Francis X. Diebold Diebold-Mariano Test _Francis X. Diebold’s Blog_fxdiebold.blogspot.nl No Hesitations In general, I stay aware of the latest developments in statistics, machine learning, data science, and other related subjects. I read a lot of new articles from scientific journals and data science mailing lists. Dogs like snow, too Quick question: Do you know the ? difference between a dog and a wolf Anyway, you get the point. If you’re going to learn something, learn it the correct way. Machines should, too. Hopefully, Evgeniy’s interview has shown you the importance of learning and using the data you have to reach true solutions, especially in this amazing digital age. This way, you’ll always know where the sun shines and animals play.

Dogs, Wolves, Data Science, and Why Machines Must Learn Like Humans Do

For more content like this, follow & subscribe to our . ButterCMS on Twitter blog As a CTO it’s important to continuously be increasing your knowledge and network. One great way to do that is to attend technology conferences in your space. Below we highlight some of the top events that CTOs should consider attending for the remainder of 2017 and the start of 2018. The events below are listed in proximity to the actual date of the event. If you’re last minute, then start at the top. 1. This is a developer-oriented conference that explores the IoT and all of its associated technologies, like mobile, web, and the cloud. It’s also held at a and has plenty of family-oriented activities, so feel free to bring the family. That Conference Wisconsin waterpark Dates: August 7th-9th 2. This is a team-oriented event, so you may want to bring a few of your developers along. You’ll learn the skills you need to boost productivity, improve testing, and help you build better . There are both classes and topics covered, so you can build your own customized week of training. Software Testing Training Week software Dates: August 21–25 3. Do you need to up your knowledge of big data, the cloud, and security? This conference explores the latest trends in the above topics, and combines equal parts theory with practical application. If big data plays in role in the future of your company, then this event could be hugely beneficial. International Conference of ICT Dates: August 21–25 4. Jupyter Notebooks are a popular and powerful tool for sharing and communicating data analysis. Open source tools like this can help you solve “last mile” issues, and help to transform your existing workflows. If you’ve been thinking about implementing Jupyter into your development cycle, this conference is worth attending. JupyterCon New York City 2017 Dates: August 23–25 5. As a CTO, you may have an overlapping role as a CIO. But, even if you don’t, having a greater understanding of the latest developments in the IT space can be beneficial. This conference explores the future of the IT and CIO space. Gartner CIO & IT Executive Summit Cancun 2017 Dates: August 29–31 6. Having a deep understanding of data and analytics can help to transform your organization. This conference will help you gain a better understanding of data in your organization, highlight how to build a data-driven culture, and much more. Chief Data and Analytics Officer Melbourne 2017 Dates: September 4–6 7. This two day event is focused solely around data design and analysis. If you’re looking to better understand your data and improve your data literacy, then you’ll want to check out this conference. Data Visualization Summit Boston 2017 Dates: September 6–7 8. This conference will teach you best practices and techniques in application development and delivery, including things like microservices, DevOps, cloud and containers. nginx.conf 2017 Dates: September 6–8 9. This conference is all about analyzing the latest trends in product management. It can be easy to get caught up in doing without thinking about the bigger picture. This conference forces you to step back, get inspired, and really think about your own product. Industry Cleveland 2017 Dates: September 13–15 10. This conference explores the rapid escalation of security, privacy, and risk from an enterprise level perspective. Learn how you can implement better security practices to ensure long-term customer trust. Forrester Privacy and Security 2017 Dates: September 14–15 11. AI is going to shake up a lot of industries. This conference focuses on implementing AI in real-world projects, as well as its potential in industries yet to be disrupted by AI technology. It brings the top minds together to cut through the hype and deliver the most practical concepts, right now. O’Reilly Artificial Intelligence Conference San Francisco 2017 Dates: September 18–20 12. This conference focuses on the place where business and technology meet, and the collaboration tools and platforms that enable the process. If you’re looking to improve your collaboration across all existing channels, then this is the conference for you. Enterprise Business Collaboration Berlin 2017 Dates: September 18–20 13. This conference brings together senior executives who are responsible for implementing and optimizing their mobility strategies and solutions. Expect to gain new insights from top executives in the field. Transformation Exchange London 2017 Dates: September 19–20 14. This is a useful event if you’re looking to shake up your customer experience strategy. Speakers have been chosen to help you gain insights into the evolving digital landscape. You’ll learn about the latest innovations in mobile, social, AI, VoC, big data, and personalization. 4th Annual Digital Customer Experience Strategies Summit Dates: September 19–20 15. This conference is dedicated to the future of work. Part of being a CTO is team management and this will help you think about how to take your team into the future. WORKTECH17 Singapore Dates: September 19 16. This conference covers the latest trends surrounding product development and overall product lifecycle. It covers things like design thinking, the role of big data, rapid prototyping and deployment, identification and execution of new opportunities and more. Product Innovation Summit Boston 2017 Dates: September 25–26 17. The Driving Change Conference dives into new and innovative approaches to cultural, behavioral, and organizational change, which is a must-have for any CTO. It will cover topics like building long-term change goals, inspiring long-term staff engagement, behavior shifts and mindsets to help embrace change and much more. Driving Change Conference London 2017 Dates: September 27 18. STARWEST is one of the longest running conferences on software testing and quality assurance. Topics to be covered at this event include testing techniques, DevOps, leadership, project management, agile testing, and more. STARWEST Software Testing Conference Anaheim 2017 Dates: October 1–6 19. This conference will highlight the most innovative advancements and technologies that are affecting IoT. It’s part exhibition and part conference, for those looking to be informed on the latest advancements in this space. Business Meet on Cloud and Internet of Things 2017 Dates: October 2–3 20. DX Day will be all about how leading UK brands across eCommerce, retail and finance are managing their digital experiences. It highlights both the business and technical issues surrounding enterprise-scale projects. DX Day London 2017 Dates: October 3 21. This conference brings together leads across every part of the technology sector to show you how to thrive in today’s digital world. Join industry leaders like Google and IBM, in discussing topics like, collaboration, information management, and more. BoxWorks San Francisco 2017 Dates: October 10–13 22. Whether you’re a Fortune 500 company, or a startup, IoTBuild will be useful. It’s an event where business and technical experts will talk freely about how they’ve built IoT business models, applications, products, and services. The future of IoT is being built right now, will you be a part of it? IoTBuild San Francisco 2017 Dates: October 11–12 23. Crazy about content management? This is a content management system conference and expo that’s a great chance for you to learn about emerging product and network with like minded techies. If you’re a CTO in the CMS space, then this event is a must-attend. CMS Summit Orlando 2017 Dates: October 14–18 24. TechWeek holds several conferences throughout the year that focuses on innovations in technology. At each conference, they partner with entrepreneurs and investors to bring you the latest advancements and news in technology. TechWeek New York 2017 Dates: October 16–20 25. This conference offers insights for security decision makers about current best practices and the future of cyber security. If you want to learn from experts about how you can build stronger defense and quickly recover from attacks, then this is the conference for you. Cyber Security Chicago 2017 Date: October 18–19 26. This innovation summit is hosted by the research firm, Constellation Research. You’ll learn from technology visionaries, market leads, and see new product demos. Learn from the best and brightest in the technology space and where it’s headed into the future. Constellation’s Connected Enterprise 2017 Dates: October 25–27 27. This summit brings together some of the world’s leaders in this industry. From programmers to experience front-end developers, this summit will help you learn how to both visualize and integrate data into your company. Data Visualization Summit London 2017 Dates: November 2–3 28. Dare Mighty Things will change the way you see the world, and was voted one of the “8 Top Tech Conferences You Can’t Miss”, by Inc.. Learn from tech leaders, entrepreneurs, and venture capitalists who share their stories on creating breakthrough innovations. Dare Mighty Things Chicago 2017 Dates: November 3 29. This conference began back in 2010 with the idea to connect the technology community with all industries. If you’re looking to open your mind and see how other industries are utilizing and contributing to the advancement of technology, then this is worth attending. Web Summit Lisbon 2017 Dates: November 7–9 30. At this conference you’ll learn fresh and innovative approaches for capturing data insights and using them to make better business decisions, and drive performance. Collecting data isn’t enough, you need a thoughtful methodology for making that data actionable. The Implementing Actionable Data Insights Conference — Drive Performance London 2017 Dates: November 14 31. The Business Show is a free exhibition that provides a wealth of business motivation and knowledge. This event features business owners from a variety of industries that centers around expanding your business. The Business Show London 2017 Dates: November 16–17 32. This is the leading conference hosting by Amazon Web Services for the global cloud community. If you’re looking for ways you can utilize AWS effectively, then this learning-based conference is for you. AWS re:INVENT Las Vegas 2017 Dates: November 27 — December 1 33. This summit will focus on the developing mobile landscape and how to most effectively utilize it in your business. It offers strategies from the leaders of mobile and how they are using mobile within their existing companies. Mobile Innovation Summit San Francisco 2017 Dates: December 4–5 34. This conference will provide you with the latest research, development and trends in information security. At this conference you’ll find a combination of research presentations and hands-on trainings. Black Hat Europe 2017 Dates: December 4–7 35. Your customer can be your biggest detractor or your biggest supporter. This conference focuses on the idea that the future belongs to companies who develop customer advocates. It seeks to teach you how to both develop and mobilize advocates. Advocamp San Francisco 2017 Dates: December 6–8 36. AI World is the industry leading event on the current state of enterprise AI and machine learning. It’s goal is to help technology executives cut through the hype and learn to deploy these advanced technologies to build competitive advantage, create new business opportunities and more. AI World 2017 Dates: December 11–13 37. As organizations change and continue to embrace technological advances, data becomes the key factor in how companies can gain a competitive advantage. This conference explores how companies can better embrace data and use it to push forward. Big Data Innovation Summer Las Vegas 2018 Dates: January 24–25 38. Most companies today are built on the back of their domain names. This conference focuses on the explosive growth of new TLDs and the expansion of internet connectivity worldwide. NamesCon Las Vegas 2018 Dates: January 29–31 39. This conference is San Francisco’s largest development conference. It includes mixers, workshops, drink-ups, and events across the city. It covers new technologies including HTML5, Node.js, Data Science, Full-Stack Javascript Development, and more. Developer’s Week San Francisco 2018 Dates: February 3–8 40. This two-day event includes two days of educational presentations, panel sessions, interactive discussions, and networking. It covers topics including data analytics, data science, predictive analytics, machine learning and algorithms, and cloud computing. Big Data & Analytics Innovation Summer Melbourne 2018 Dates: February 7–8 41. This summit provides a platform for industry leaders to deliver case studies, expertise and unique insights into the power of being able to forecast the future with today’s data. Learn how practitioners deploy predictive modeling, and its impact on business. Innovation Summit San Diego 2018 Dates: February 14–15 42. If your company is involved in the mobile landscape, then you’ll find this conference valuable. This conference dives deep into new and emerging mobile technologies and how they’re going to disrupt the mobile landscape. Mobile Beat Las Vegas 2018 Dates: March 12–15 43. This conference brings together the World of Watson and Interconnect to discuss the future of AI, cloud, data, IoT and security. AI is already causing stirs in all sorts of industries, so by the time 2018 rolls around, you’ll want to see what it’s up to, and this event is the way to do that. Think Las Vegas 2018 Dates: March 19–22 Check That Calendar If you want to keep growing as a CTO, check that calendar, clear a few dates with the boss, and purchase those tickets — because you’ve got some conferencing to do! Know of any other conferences for CTOs coming up? Share them with us in the comments below. Originally published at buttercms.com . For more content like this, follow ButterCMS on Twitter and subscribe to our blog .

Stories, News & Mentions about PETERPARK

Peterpark’s Stories on HackerNoon