Shopgram

shopgram

Hello everyone, dev.family is in touch. We would like to tell you about an interesting project that we have been working on for almost six months and are still continuing. During this time, a lot has happened in it, a lot has changed. We discovered something interesting for ourselves, managed to fill the bumps. What will our story be like? What did we do How we started Where did this lead us What problems did we face Why monorepo Why PNPM Why TS How it works now How much we have made our lives easier A little about the project So, what are we still working on? In fact, this question at some point became very relevant, as it was, for example, for the owner of McDonalds corporation at one time. We started the project as a crypto-loyalty program that provides end-users with rewards for certain actions, and customers receive analytics on these same users. Yes, it's pretty superficial, but it doesn't matter. Beginning of work It was necessary to develop Shopify modules for connecting to Shopify stores, a portal for brands, an extension for Google Chrome, a mobile application + a server with a database (well, actually, nowhere without them). In general, with what we need, we decided and started working. Since the project was immediately assumed to be large, everyone understood that it could grow like magic beans of delayed action. It was decided to do everything “correctly” and “by all standards”. That is, everything is written in one language - TypeScript. So that everyone writes the same way, and there are no unnecessary changes in the files, linters (a lot of linters), so that everything is “easy” to reuse, put EVERYTHING into separate modules, and so that they do not steal under the Github access token. So we started: Repository for linters and ts config separate (style guide) Repository for a mobile application (react native) and a Chrome extension (react.js) (together, since they repeat the same functionality, only aimed at different users) Another repository for the portal Two repositories for Shopify modules Repository for blockchain stuff API repository (express.js) Repository for infrastructure Huh ... think I listed everything. It turned out a bit too much, but okay, let’s keep rolling. Oh yeah, why were two repositories allocated for Shopify modules? Because the first  repository is UI-modules. There is all the beauty of our babies and their settings. And the second is integrations-Shopify. This is in fact its’ implementation in Shopify itself  with all liquid files. In total, we have 8 repositories, where some should communicate with each other. Since we are talking about development in TypeScript, we also need package managers to install modules, libraries. But we all worked independently in our repositories, and it didn't matter to anyone what to use. For example, while developing a mobile application on React Native, I did not think for too long and kept YARN1. Someone may be more used to using the good old NPM, while others love everything new and use the fresh YARN3. Thus, somewhere there was NPM, somewhere YARN1, and somewhere YARN3. So we all started to make our applications. And almost immediately the fun began, but not that complete. Firstly, some did not think about what TypeScript was for, and used “Any” wherever they were too lazy, or where they “didn’t understand” how they could not write it. Someone did not realize all TypeScript power and the fact that in some places everything can be made much easier. Therefore, the types came out of cosmic dimensions. Yes, I forgot to say, we decided to use Hasura GraphQL as a database. Manual typization of all answers from it sometimes looked like something else. And in one case, some even wrote in good old Javascript. Yes, the situation turned out to be cool: the first guy put “Any” once again to not to strain too much, the second writes canvases of types with his own hands, and the third still do not write types at all. Later it turns out that in cases where we repeated the logic, and, in a good way, it should have been taken out in a separate package - no one was going to do this. Everyone writes and writes code for himself, for everything else - spit from a high bell tower. Where did this lead us? What do we have? We have 8 repositories with different applications. Some are needed everywhere, others communicate with each other. Therefore, we all create .NPMrc files, prescribe credits, create a github token, then through the package manager-module. In general  a slight hassle, although unpleasant, but nothing unusual. Only in the case of updating something in the package, you need to upgrade its version, then upload it, then update it in your application/module, and only then you will see what has changed. But this is totally inappropriate! Especially if you can just change the color somewhere. In addition, some code is repeated and not reused, but simply quietly rewritten. If we are talking about a mobile application and a browser extension, the redux store and all the work with the API are completely repeated there, something is just completely rewritten or slightly modified. In total, what we are left with: a bunch of repositories, a rather long launch of applications / modules, a lot of the same things written by the same people, a lot of time spent on testing and introducing new people into the project, and other problems arising from the above. In short, this led us to the fact that the tasks were performed for a very long time. Of course, this led to deadlines being missed, it was quite difficult to introduce someone new to the project, which once again affected the speed of development. Everything was going to be quite dreary and long, in some cases, thanks to webpack for that. Then it became clear that we were moving far from where we were striving, but who knows where. After analyzing all the mistakes, we made a number of decisions, which will be discussed now. Why monorepo? Probably, the most important thing that influenced a lot in the future was the realization that we are not building a specific application, but a platform. We have several types of users, there are different applications for them, but they operate within the same platform. So we immediately closed the issue with a large number of repositories: if we are working on one platform, why split it into repositories when it is easier to work in one. I want to say that working in a monorepo made our lives a hell of a lot easier. Some applications or modules had a direct relationship with each other, and now you can work on them with peace of mind on the same branch in the same repository. But this is far from the main advantage. Let's continue. We have moved everything into one repository. Cool! We continued to work at the same pace until it came to reusability. In fact, this is a “rule of good taste” we have in our work. Realizing that in some places we use the same algorithms, functions, code, and in some places separate packages that we installed via github, we decided that all this “doesn’t smell very good” and started putting everything into separate packages within a monorepo using Workspaces. Workspaces (workspaces) are sets of functions in the NPM cli, with which you can manage multiple packages from a single top-level root package. In fact, these are packages within one package that are linked through a specific package manager (any YARN / NPM / PNPM), and then used in another package. To tell the truth, we did not immediately rewrite everything on workspaces, but did it as needed. Here's what it looks like: From one file { "type": "module", "name": "package-name-1", ... "types": "./src/index.ts", "exports": { ".": "./src/index.ts" }, }, To another file { "type": "module", "name": "package-name-2", ... "dependencies": { "package-name-1": "workspace:*", }, }, An example using PNPM Nothing complicated, actually if you think about it: write a couple of commands and lines, and then use whatever you want and wherever you want. But “there is one caveat, comrades”. Earlier I wrote that everyone used the package manager they wanted. In short, we have a repository with different managers. In some places it was funny when someone wrote that he could not link this or that package, having in mind a fact that he uses NPM, and there is YARN. I will add that the problem was not because of different managers, but because people used the wrong commands or configured something wrong. For example, some people through YARN 3 just did a YARN link and that's it, but for YARN 1 it didn't work the way they wanted due to the lack of backward compatibility. After switching to monorepo Why PNPM? By this point, it became clear that it is better to use the same package manager. But you need to choose which one, so at that time we considered only 2 options - and . We discarded NPM right away, because it was slower than others and uglier. There was a choice between PNPM and YARN. YARN PNPM YARN initially worked well - it was faster, simpler and more understandable, which is why everyone used it then. But the person who made YARN left Facebook, and the development of the next versions og it was transferred to others. This is how YARN 2 and YARN 3 appeared without backward compatibility with the former. Also, in addition to the yarn.lock file, they generate a yarn folder, which sometimes weighs as node_modules and stores caches in itself. Therefore, we, like many other developers, turned our attention to PNPM. It turned out to be as convenient as the first YARN at its time. Workspaces can be easily used here, some commands look the same as in the first YARN. In addition, shamefully-hoist turned out to be a nice additional option - it is more convenient to install node_modules everywhere at once than go to some folder every time and do PNPM install. Turborepo and code reuse In addition, we decided to try turborepo. Turborepo is a CI/CD tool that has its own set of options, cli, and configuration via turbo.json file. Installed and configured as easy as possible. We put a global copy of the turbo cli through PNPM add turbo --global. Adding turbo.json to the project turbo.json { "$schema": "https://turbo.build/schema.json", "pipeline": { "build": { "dependsOn": ["^build"] } } } After that we can use all the available functions of turborepo. We were most attracted by its features and the possibility of using it in a monorepo. What hooked us: Incremental builds (Incremental builds - collecting builds is quite painful, Turborepo will remember what was built and skip what has already been calculated); Content-aware hashing (Content-aware hashing - Turborepo looks at the contents of files, not at timestamps, to figure out what needs to be built); Remote Caching (Remote hashing - share a remote build cache with the team and CI / CD for even faster builds.); Task pipelines (A task pipeline that defines relationships between tasks and then optimizes what and when to create.). Parallel execution (Performs builds using each core with maximum parallelism, without wasting idle CPUs). We also took the recommendation for organizing a monorepo from the documentation and implemented it in our platform. That is, we split all our packages into apps and packages. To do this, we also create the PNPM-workspace.yaml file and write: PNPM-workspace.yaml packages: 'apps/**/*' 'packages/**/*' Here you can see an example of our structure before and after: Now we have a monorep with customized workspaces and convenient code reuse. I will add a few more points that we did in parallel. I mentioned two things earlier: we had a chrome extension, and that we decided - we were making a platform. Since our platform worked with Shopify as a priority, we decided that instead of an extension for Chrome or in addition to it, it would be nice to make another module for Shopify, which can be simply installed on the site, so as not to once again force people to download a mobile application or Chrome extension . But it must completely repeat the extension. Initially, we did them in parallel, but we realized that we were doing something wrong, because we simply duplicated the code. In every sense we write the same thing in different places. But since we now have all the workspaces and reuse configured, we easily moved everything into one package, which we called in the Shopify module and Chrome extension. Thus, we saved ourselves a lot of time. The second thing that saved us a lot of time was the elimination of webpack and, in some places, builds in general. What's wrong with webpack? In fact, there are two critical points: complexity and speed. What we have chosen is vite. Why? It is easier to set up, it is quickly gaining popularity and already has a large number of working plugins, and an example from the docks is enough for installation. In comparison, the build on the webpack of our Chrome web extension took about 15 seconds, on vite.js about 7 seconds (with dts file generation). Feel the difference. What's with the rejection of builds? Everything is simple, as it turned out, we didn’t really need them, since these are reusable modules and in package.json, in exports, you could simply replace dist/index.js with src/index.ts. How it was {... "exports": { "import": "./dist/built-index.js" }, ... } How it’s now { ... "types": "./src/index.ts", "exports": { ".": "./src/index.ts" }, ... } Thus, we got rid of the need to run PNPM watch to track application updates related to those modules, and do PNPM build to pull updates. I don't think it's worth explaining how much time it saved us. In fact, one of the reasons why we collected builds was TypeScript, more precisely index.d.ts files. So that when importing our modules / packages, we know what types are expected in certain functions or what types others will return to us, such as here: But given that you can simply export from index.tsx, there was another reason to abandon builds. TypeScript + GraphQL But still, why TypeScript? I think it makes no sense now to describe all the advantages of TS: type safety, facilitating the development process due to typing, the presence of interfaces and classes, open source code, errors made during code modification are visible immediately, and not at runtime, and so on. As I said at the very beginning, we decided to write everything in one language so that if someone stops working or leaves, we can support or insure. First we chose JS. But JS is not very secure, and without tests on large projects it is quite painful. Therefore, we decided in favor of TS. As practice has shown, it is very convenient in monorepo, due to the fact that you can simply export *.ts files, and when using components, the expected data and their types are immediately clear. But one of the main useful features was auto-generation of types for GraphQl query and mutations. For everyone who is not very knowledgeable, GraphQl is a technology that allows you to go to the database through the same query (to get data) and mutation (to change data), and looks something like this: query getShop {shop { shopName shopLocation } } Unlike the REST API, where until you receive it, you won’t know what will come to you, here you yourself determine the data that you need. Let's get back to our President-elect. We used Hasura, which is a GraphQL wrapper on top of PostgreSQL. Since we are working with TS, then in a good way we must type the data from both requests and those that we send to payload. If we are talking about the code from the example above, there should be no problems, kind of. But in practice, a query can reach a hundred lines, plus some fields may or may not come, or have different data types. And typing such canvases is a very long and thankless task. Alternative? Of course I have! Let the types be generated via commands. In our project, we did the following: We used the following libraries: graphql and graphql-request First, files with *.graphql resolution were created, in which queries and mutations were written. For example: test.graphql query getAllShops {test_shops { identifier name location owner_id url domain type owner { name owner_id } } } Next we created codegen.yaml codegen.yaml schema: ${HASURA_URL}:headers: x-hasura-admin-secret: ${HASURA_SECRET} emitLegacyCommonJSImports: false config: gqlImport: graphql-tag#gql scalars: numeric: string uuid: string bigint: string timestamptz: string smallint: number generates: src/infrastructure/api/graphQl/operations.ts: documents: 'src/**/*.graphql' plugins: - TypeScript - TypeScript-operations - TypeScript-graphql-request There we indicated where we were going, and at the end - where we save the file with the generated API (src/infrastructure/api/graphQl/operations.ts) and where we get our requests from (src/**/*.graphql). After that, a script was added to package.json that generated the same types for us: package.json {... "scripts": { "generate": "HASURA_URL=http://localhost:9696/v1/graphql HASURA_SECRET=secret graphql-codegen-esm --config codegen.yml", ... }, ... } They indicated the URL that the script accessed to obtain information, the secret, and the command itself. Finally, we create the client: import { GraphQLClient } from "graphql-request"; import { getSdk } from "./operations.js"; export const createGraphQlClient = ({ getToken }: CreateGraphQlClient) => { const graphQLClient = new GraphQLClient(‘your url goes here...’); return getSdk(graphQLClient); }; Thus, we get a function that generates a client with all query and mutations. The bonus in operations.ts lays all our types that we can export and use, and there is a complete typing of the entire request: we know what needs to be given and what will come. You don't need to think about anything else, except to run the command and enjoy the beauty of typing. Conclusion Thus, we got rid of a large number of unnecessary repositories and the need to constantly push the slightest changes in order to check how things work. Instead, they came up with one in which everything is structured, decomposed according to its purpose, and everything is easily reused. So we made our life easier and reduced the time to introduce new people to the project, to launch the platform and modules / applications separately. Everything has been typed, and now there is no need to go into each folder and see what this or that function/component wants. As a result, development time has been reduced. In conclusion, I want to say that you should never be in a hurry. It is better to understand what you are doing and how to do it easier than to deliberately complicate your life. Problems are everywhere and always, sooner or later they will come out somewhere, and then deliberate complication will shoot you in the knee, but will not help in any way. The dev.family team was with you, see you soon!

Monorepository in TypeScript: The Story of How we Broke Everything and Made it Better

Monorepositorio en TypeScript: la historia de cómo rompimos todo y lo mejoramos

Monorepository dans TypeScript : l'histoire de la façon dont nous avons tout cassé et amélioré

टाइपस्क्रिप्ट में मोनोरेपोसिटरी: द स्टोरी ऑफ़ हाउ वी ब्रोक एवरीथिंग एंड मेड इट बेटर

TypeScript のモノレポジトリ: すべてを壊してより良くする方法の話

Monorepositório em TypeScript: a história de como quebramos tudo e melhoramos

Kho lưu trữ đơn trong TypeScript: Câu chuyện về cách chúng tôi phá vỡ mọi thứ và làm cho nó tốt hơn

TypeScript 中的单一存储库：我们如何打破一切并使其变得更好的故事

Data will talk to people if you can present it elegantly. The world of web development is adorned with plenty of amazing libraries for charts, but do we have some for mobile apps too? Is there a library with which you can- react-native Draw all kinds of charts? (Bar, Line, Area, Pie, Stack) Add gradient and 3D effects with a snap of a finger? Fully customize your charts? Add smooth animations, without writing actual codes for animation? Add onPress events to each data component? Scroll through your data, as it’s too long to fit the mobile’s screen width? That needs minimal training and has excellent documentation? react-native-gifted-charts caters to all the above needs. You can install it from npm- npm install react-native-gifted-charts react-native-linear-gradient react-native-svg Github repo- https://github.com/Abhinandan-Kushwaha/react-native-gifted-charts It’s simple and intuitive. Let’s say you want to draw a Bar chart with data- [15, 30, 26, 40] The data you provide to the chart component looks like this- barData = [{value: 15}, {value: 30}, {value: 26}, {value: 40}]; And the code for this chart is - import { BarChart } from "react-native-gifted-charts";
        
const App = () => {
    const barData = [{value: 15}, {value: 30}, {value: 26}, {value: 40}];
    return <BarChart data={barData}/>;
}; Now you wish to change the colour of the bars. Just add to the component. frontColor=‘#color-code’ <BarChart> Simple, isn’t it? Not just you can configure a whole list of properties, simply by using props like these- color, barWidth showGradient gradientColor roundedTop roundedBottom activeOpacity disablePress barBorderRadius In case, you want a 3D effect, these props will help you- isThreeD barWidth sideWidth sideColor topColor Okay, I’m not here to scare you with a humongous list of props. I’d rather share with you the complete . cheat sheet of props Everything seems fine up to here. But my designer gave me a chart that looks like this- The data for this chart is- [500, 745, 320, 600, 256, 300] This scares me because if I use the prop, it changes the color of each bar. How do I change the color of some specific bars? 😱 frontColor Soon I figured out that I could pass the along with the value in the data array. Also, there’s a label (weekday) under each bar, so I need to pass the too. color label So, the data array now looks like- const data = [
        {value: 250, label: 'M'},
        {value: 500, label: 'T', frontColor: '#177AD5'},
        {value: 745, label: 'W', frontColor: '#177AD5'},
        {value: 320, label: 'T'},
        {value: 600, label: 'F', frontColor: '#177AD5'},
        {value: 256, label: 'S'},
        {value: 300, label: 'S'}
]; Wow! it’s so intuitive. 😲 I suppose you’ve already guessed that there are 2 ways to pass the props- Directly as a prop to the component. <BarChart> As a property of the object in the data array. This lets you apply the properties selectively to some specific bars. On , you can find codes for beautiful charts like these- this link Stacked Bar Charts One day I saw something like this- I’m sorry, whaaaat?? Are they making charts like this? Anyways, let’s see what gifted-chart has to say on this. Okay, so here we have stacks of bars for each month. That means the data is definitely a 2D array, somewhat like this- [ 
  [10, 20],
  [10, 11, 15],
  [14, 18],
  [7, 11, 10]
] Understanding the stackData The stackData passed to the component is an array of objects. <BarChart> Each object contains a mandatory key named . stacks The value corresponding to the key is an array of objects, each object representing a section of the stack. stacks The actual code for the above chart is- import { BarChart } from "react-native-gifted-charts";
        
const App = () => {
    const stackData = [
        {
          stacks: [
            {value: 10, color: 'orange'},
            {value: 20, color: '#4ABFF4', marginBottom: 2},
          ],
          label: 'Jan',
        },
        {
          stacks: [
            {value: 10, color: '#4ABFF4'},
            {value: 11, color: 'orange', marginBottom: 2},
            {value: 15, color: '#28B2B3', marginBottom: 2},
          ],
          label: 'Mar',
        },
        {
          stacks: [
            {value: 14, color: 'orange'},
            {value: 18, color: '#4ABFF4', marginBottom: 2},
          ],
          label: 'Feb',
        },
        {
          stacks: [
            {value: 7, color: '#4ABFF4'},
            {value: 11, color: 'orange', marginBottom: 2},
            {value: 10, color: '#28B2B3', marginBottom: 2},
          ],
          label: 'Mar',
        },
      ];
    return(
        <View>
            <BarChart
              width={340}
              rotateLabel
              barWidth={12}
              spacing={40}
              noOfSections={4}
              barBorderRadius={6}
              stackData={stackData}
            />
        </View>
      );
}; Line and Area charts Let’s begin with the simplest Line chart visualising the same data that we used in our first Bar chart- [15, 30, 26, 40] Since the data is the same, we can use the same code, just replace the component with <BarChart> <LineChart> import { LineChart } from "react-native-gifted-charts";
        
const App = () => {
    const data = [{value: 15}, {value: 30}, {value: 26}, {value: 40}];
    return <LineChart data={data}/>;
}; You can convert any line chart into area chart by just passing the prop to the <LineChart/> component. Pro tip: areaChart On , you can find codes for beautiful Line charts and Area charts like these- this link Pie and Donut charts Can we use the same code, as our first example? Let’s see- import { PieChart } from "react-native-gifted-charts";
        
const App = () => {
    const data = [{value: 15}, {value: 30}, {value: 26}, {value: 40}];
    return <PieChart data={data}/>;
}; Only the component is replaced with and it produces this- <BarChart> <PieChart> I know it looks shitty, and that’s probably because we didn’t provide any colors for the pie sections and the library tried to give random colors by itself. But I swear you can make beautiful ones like these- Want the codes? they are. Here And that’s it for now!

How to Make Beautiful, Easy-to-read Charts with React Native Gifted Charts

Apart from multiple technological developments influencing the retail market, there is one that creates a connection between all of them. ECommerce API is gradually boosting eCommerce websites by allowing them to communicate with one another. It is possible to utilize eCommerce API for data transmission between software, so then it can be used from one source. API integration tools serve as extensible platforms that interface with a variety of solutions, allowing businesses to leverage their existing capabilities rather than having to design new features from start. ECommerce API delivers a variety of crucial advantages for competitiveness in the retail market, including omnichannel, new digital channels, real-time monitoring, and context-based offers. There is a possibility to link with partners and eCommerce websites that deploy APIs, provide their products through these companies’ digital channels, and vice versa. Since such API integration tools need to handle a large number of requests, responses, and logic at the same time, it is better to implement one of the giants on the market, for example, WooCommerce API or Shopify API. Both platforms enable the development of a bespoke online store, even though they have . While making a decision on which one to choose for your online store: WooCommerce API or Shopify API, figure out if the setup process also fits your expectations. For your convenience, below you will find a gradual guide on eCommerce API integration for WooCommerce and Shopify. differences and drawbacks Integrating Shopify API Shopify is an all-in-one solution that lets you create an online store. With a few clicks, you can organize your products, customize your storefront, accept credit card payments, track and respond to orders. With the implementation of Shopify API, you can ensure payment security. Shopify’s technology is built to be simple to use. You can get an all-in-one eCommerce API that’s ready to go as soon as you are. Getting started In order for your app to access the Shopify API the first step should be creating an application within your Shopify partners account. After you have signed up, in your dashboard select the Apps section and create the app to integrate Shopify API. Select a public app if you want to integrate different Shopify stores. In general, settings fill in your app name, app URL and provide a redirect URL, which will handle successful authorization to Shopify account in your app. In our case, it’s our backend endpoint. After the app is created, we can get our API keys. The keys are suitable for every store based on Shopify API. OAuth To start the OAuth process we should open the OAuth window for a specific store. The URL has the format: https://{shop}.myshopify.com/admin/oauth/authorize?client_id={api_key}&scope={scopes}&redirect_uri={redirect_uri}&state={nonce}&grant_options[]={access_mode} ***shop***: the name of the shop, 

            ***scopes***: list of the API resources to be accessed, 

            ***nonce***: can be anything, we can put there the user token of our app as an example (it                    will be returned to our handler), 

            ***access_mode***: can be ‘per_user’ for online access, and ‘value’ for offline access. In our case the OAuth URL will be the following: https://test-incora.myshopify.com/admin/oauth/authorize?client_id=705bea4b9fa544fa57008b48ff2d0228&scope=read_customers,write_products&redirect_uri=http://localhost:8080/shopify-auth&state=user_token&grant_options[]=per_user After you successfully logged in to your Shopify account you’ll see the page where you should confirm app installation. After confirming app installation you will be redirected to the URL you specified. As we referenced our back-end endpoint, let’s create one. The payload from Shopify will include the code, which can be exchanged to get access tokens for the listed scopes. Let’s add the method to exchange the code. We are using Axios as an HTTP client. const getToken = (shopname, data) => {
  return axios.post(`https://${shopname}/admin/oauth/access_token`, data);
}; The request handler should look like below. router.get('/shopify-auth, async (req, res) => {
 const { code, shop } = req.query;
 const { data: { access_token: token } } = await getToken(shop, {
      client_id: /* api_key */,
      client_secret:  /* api_secret */,
      code,
 });

  // save token to e.g. redis
  // redirect to the FE page
}); Our query params payload will contain code that will only be used once to obtain an access token. It’s better to save a token to some cache storage and send a redirect as the response. Using API After you get the token you can use the endpoints of Shopify API. For example, let’s add a method to get products of the shop: const getProducts = (shopname, token, params = {}) => {
  return axios.get(`https://${shopname}/admin/api//2021-01products.json`, {
    params,
    headers: { 'X-Shopify-Access-Token': token }
  });
}; At this step, the installation process is over, and you are good to go with applied API integration tools, that will enable your online store to fully functioning. Applying Woocommerce API WooCommerce is an open-source eCommerce plugin for WordPress. It is designed for small and large-sized online merchants using WordPress. It simplifies the process of setting up and running an online store by providing respectable levels of flexibility and a number of essential features such as inventory and tax administration, secure payment processing, and shipping integration. The WordPress REST API is strongly embedded with the WooCommerce API. This enables requests in , WordPress REST API Authentication methods, and regular HTTP verbs to be used to create, read, update, and delete its contents. JSON format To start API integration, you should prepare for the 4 main steps: Authorization, Import, Export, and Webhooks. Let’s begin with Authorization. Authorization We use a consumer key, consumer secret, and host (WooCommerce shop name) for all requests to the WooCommerce API. const updateWoocommerceConnectionField = async (req, res) => {
 try {
   const userId = req.user.id;
   const { consumerKey, consumerSecret, host } = req.body;
   await woocommerceHelper.checkKeys(host, consumerKey, consumerSecret);
   await User.query()
     .where({ id: userId })
     .update({
       woocommerce_connection: {
         consumerKey,
         consumerSecret,
         host: formatHost(host),
       }
     });
   const params = { consumer_key: consumerKey, consumer_secret: consumerSecret };
   createWebHook(host, params);
   return res.status(200).json();
 } catch (e) {
   apiHelper.builder(res, getError(e));
 }
}; So to get the consumer key and secret you need to create an account on WooCommerce, then login to the admin page and press . On this page you should click “Add key” and here you can choose the level of access for this REST API key, which can be Read access, Write access or Read/Write access. “Woocommerce”-> “Settings”-> “Advanced”-> “REST API” After you’ve completed these steps, click the “Generate API Key” button, and the WooCommerce API will create REST API keys for the user you specified. You can see these credentials only one time when you generate them, so you should save them. API keys are not transferred to other users, it means one user – one consumer secret and key. Import Now our app has three integrations with Shopify, Woocommerce, and Magento, so we use the Factory method to initialize new integration services. class IntegrationServiceFactory {
 static create(serviceName) {
   switch (serviceName) {
     case INTEGRATIONS.SERVICES_NAMES.SHOPIFY:
       return new ShopifyIntegrationsService(serviceName);
     case INTEGRATIONS.SERVICES_NAMES.WOOCOMMERCE:
       return new WoocommerceIntegrationsService(serviceName);
     case INTEGRATIONS.SERVICES_NAMES.MAGENTO:
       return new MagentoIntegrationsService(serviceName);
     default:
       return null;
   }
 }
} To find the products which are necessary, we employ pagination. Woocommerce REST API offers a lot of different parameters for pagination such as offset, search, per_page, order, orderby. For every request, you should add a consumer key and secret to the parameters. The first-order position in every endpoint should be the host (your shop name). The current version of the WP REST API integration is v3, which takes a second-order position, and the product ID is the last part of the URL. So, the request should look like this: curl https://host.com/wp-json/wc/v3/products/794 \
    -u consumer_key,consumer_secret Please note: The total number of resources and pages are always included in the X-WP-Total and X-WP-TotalPages HTTP headers. Export Export is not as easy as an import. We need some more requests and steps to create the products at WooCommerce. There are three parts: setting data for synchronization, creating, and updating. When we talk about synchronization it is a good practice to have the table at DB (ProductsSources at our app) where already imported or exported products will be stored. We filter the products which do not exist at WooCommerce by using our ProductsSources and then create them there. async createProducts() {
   const productsToCreate = this.products.filter((pr) => {
     const source = this.productsSources
       .find(ps => Number(ps.productId) === Number(pr.id));
     return !source || !this.thirdPartyProducts
       .some(({ id }) => Number(id) === Number(source.sourceData.id));
   });
   const createdProducts = [];
   for (const product of productsToCreate) {
     const result = await this.createProduct(product);
     await ProductsSources.add({
       productId: product.id,
       serviceName: this._serviceName,
       userId: this.user.id,
       sourceData: this.getSourceData(result),
     });
     createdProducts.push(result);
   }
   return createdProducts.length;
 } Send a POST request to Woocommerce API to create a product: curl -X POST https://host.com/wp-json/wc/v3/products \
    -u consumer_key,consumer_secret \
    -H "Content-Type: application/json" \
    -d '{
  "name": "Premium Quality",
  "price": "21.99",
  "description": "Lorem ipsum dolor sit amet, consectetur 
adipiscing elit, sed do eiusmod tempor incididunt ut labore et 
dolore magna aliqua. Ut enim ad minim veniam, quis nostrud 
exercitation ullamco laboris nisi ut aliquip ex ea commodo 
consequat.",
    }
  ]
}' Please note: If you want to get all products from WooCommerce you need to use an iterator because requests that return multiple items (products in our case) will be paginated to 10 items by default. When non-existing products are already created, we need to update the other products which we have at ProductSources. We look for a similar product in our app and update it at WooCommerce with our data. The update request looks like the one for creation, but you need to add a product ID at the end of the URL. WooCommerce products IDs are stored at ProductsSources. The example of a PUT request to update a product is shown below. curl -X PUT https://host.com/wp-json/wc/v3/products/228 \
    -u consumer_key,consumer_secret \
    -H "Content-Type: application/json" \
    -d '{
  "price": "22.05"
}' Webhooks Our integration needs to create, update and delete webhook functionality. So we have three routes (delivery_url) for each webhook. .post(
   '/woocommerce_order_paid',
   verifyWoocommerceWebhook([
     WOOCOMMERCE_WEBHOOK_TOPICS.ORDERS_CREATE,
   ]),
   webhooksController.woocommerceOrderCreatedController,
 )
 .post(
   '/woocommerce_order_updated',
   verifyWoocommerceWebhook([
     WOOCOMMERCE_WEBHOOK_TOPICS.ORDERS_UPDATE,
   ]),
   webhooksController.woocommerceOrderUpdatedController,
 )
 .post(
   '/woocommerce_order_deleted',
   verifyWoocommerceWebhook([
     WOOCOMMERCE_WEBHOOK_TOPICS.ORDERS_DELETE,
   ]),
   webhooksController.woocommerceOrderDeletedController,
 ) When we talk about WooCommerce authorization, you can see the function ‘createWebhook’. So we get all webhooks from WooCommerce and check if any webhook with the same topics and delivery URL doesn’t exist. We use an iterator to get all of them. Pay attention to the notes above, because they will be helpful for an iterator function either. const createWebHook = async (host, params) => {
const webhooksIterator = await new WoocommerceDataGetterIterator('webhooks', 
host, params);
 const webhooks = [];
 while (webhooksIterator.hasNext()) {
   webhooks.push(...(await webhooksIterator.next()));
 }
 for (const webhookTopic of Object.values(WOOCOMMERCE_WEBHOOK_TOPICS)) {
   if (!webhooks.some(webhook => webhook.delivery_url === 
WOOCOMMERCE_WEBHOOK_HANDLERS_URLS[webhookTopic]
       && webhook.topic === webhookTopic)) {
     const webhookCreateData = {
       name: WOOCOMMERCE_WEBHOOK_NAME[webhookTopic],
       topic: webhookTopic,
       delivery_url: WOOCOMMERCE_WEBHOOK_HANDLERS_URLS[webhookTopic],
       secret: WOOCOMMERCE_WEBHOOK_SECRET,
     };
     await woocommerceApi.createWebHooks(host, webhookCreateData, params);
   }
 }
}; So the data for this request looks like this:

                  ***name***: the name which the user can see on the Admin page 

                  ***topic***: which type of webhook you want to create delivery_url: the URL where the                            payload is delivered 

                  ***secret***: the secret key that helps to build an HMAC-SHA256 hash of the request                             body for enabling webhook’s validity. This request helps you to create a webhook: curl -X POST https://example.com/wp-json/wc/v3/webhooks \
    -u consumer_key:consumer_secret \
    -H "Content-Type: application/json" \
    -d '{
  "name": "Order create Incora",
  "topic": "order.updated",
  "delivery_url": "http://yourapi/yourpath"
"secret": "secret"
}' Please note: Woocommerce supports only HTTPS redirect URLs. So, for local testing, you will need to set up ngrok. Also, you can see “verifyWooCommerceWebhook” middleware on each of these three routes. The main functionality is to create “generatedHmac” from request body using secret, which we added to Webhook POST request. If this value is equal to x-wc-webhook-signature which will be included in request headers, it means the request came from WooCommerce REST API. Also, you can check if the topic is correct by getting it from the x-wc-webhook-topic header. const verifyWoocommerceWebhook = topics => async (req, res, next) => {
 try {
   const headerHmac = req.headers[WOOCOMMERCE_HEADERS_KEYS.WEBHOOK_SIGNATURE];
   const generatedHmac = crypto
     .createHmac('sha256', secret)
     .update(req.bufferBody)
     .digest('base64');
   if (headerHmac !== generatedHmac) {
     throw new Error(IMPOSSIBLE_VERIFYING);
   }
   if (!topics.some(topic => req.headers[WOOCOMMERCE_HEADERS_KEYS.WEBHOOK_TOPIC] 
=== topic)) {
     throw new Error(INCORRECT_TOPIC);
   }
   next();
 } catch (e) {
   apiHelper.builder(res, getError(e));
 }
}; Please note: You might find it useful to add this middleware to your delivery URLs. If there isn’t a verifying function, anyone who wants to throttle your API can send a request to these URLs. That is the whole process of API integration for WooCommerce. After fulfilling all four steps we described, your online store should be ready for the production stage. Wrapping up The ECommerce API integration can help you enhance your business processes, increase engagement, connect your customers with goods they want, and far more. No matter if you choose to implement WooCommerce API or Shopify API, either way by deploying API integration tools the e-commerce shipping and order fulfillment of your online store will be much easier. Thus, to make your existing or future online store benefit from the eCommerce API integration, you should consider deploying one of the well-known eCommerce solutions. Also posted on https://incorainc.com/ecommerce-api-in-action-woocommerce-shopify-setup-in-2021/

ECommerce APIs: How to Set Up the WooCommerce API & Shopify API

This guide will cover connecting to the with using API Keys in conjunction with the HTTP Basic scheme and some request examples. Shopify Admin API Linx authentication Linx is a low code developer tool for IT professionals. It works on any stack making use of your existing (databases, files, APIs etc). You can use it to ABOUT LINX Build and host APIs and Microservices (E.g. a large bank runs their Business Banking Customer API on Linx) Create endpoints for webhooks (E.g. this site is integrated with our marketing systems and Slack using webhooks and Linx) Integrate systems ( E.g. a mining and manufacturing supplier integrates their online shop, Salesforce and Sage X3 with Linx) Automate processes (E.g. a large investment company automates their client correspondence with Linx) Creating a Linx application and connecting it to Shopify involves the following steps: Creating a private app and generating an API Key Configuring Linx to authenticate Customizing requests RESOURCES : Developer documentation related to the Shopify Admin API. Shopify REST API reference documentation (22.5 KB) : Template Linx application. Open it in the to follow this guide. ShopifyTemplate.lsoz Linx Designer Step 1: Create a Private App To access and modify your Shopify resources using Linx, you first need to . This will then which must be used when making requests from Linx. register a private app with Shopify generate the relevant API Key To register a private app, log in to your Shopify dashboard, navigate to the Apps dashboard (Left menu > ). Click on the link which will open your private app dashboard. If you haven’t already, . Apps Manage private apps enable private app development Once enabled, click on the "Create New Private App" button. Give your app a name such as “LinxApp” and complete the contact details. In the section, show the inactive permissions of the app (by default none are selected). Admin API Select the relevant permissions, these will be the access scope of the linked application. In this guide we will be reading and writing data to the and resources, you can modify the access permissions at a later stage. Customers Products Scroll to the bottom of the page and click then . Save Create App Your app authentication details will be displayed: : Key used as the username in the HTTP Basic authentication of the request as well as the request URL. API Key : Used as the password in the HTTP Basic authentication of the request. Password : Example of the base URL to use when requesting to your instance. Base URL Step 2: Connect to Shopify To connect to the Shopify API, drag a onto a user-defined function and configure the relevant properties described below. CallRESTEndpointFNC A best practice is creating values for all the needed constants in our application that may change at a later stage. By making these $.Settings values, we can reference them throughout our Linx application and update them in a single place to have application-wide effects. $.Settings Create new values like below and add your authentication details. $.Settings Creating the Base URL When making requests to the Shopify API, you need to build up the base URL of each request like below: https: //{apikey}:{password}@{hostname}/admin/api/{version}/{resource}.json In Linx, this can be done via the use of expressions or using dynamic settings. In the provided template application the URL is built up via a dynamic setting expression. The has the structure of the URL and placeholder references which reference other values. At runtime, the values referenced in the other settings will be concatenated into a base URL. $.Settings.shopify _ baseUrl $.Settings https: //{shopify_apiKey}:{shopify_password}@{shopify_shopName}.myshopify.com/admin/api/{shopify_api_version} At runtime, the value of will be: $.Settings.shopify_baseUrl When a request is made, the base URL can just be based on this value. The particular resource path can then be added in each request using an expression like below: $.Setting = $.Settings.shopify_baseURI + + "/products" ".json" Authentication Private applications authenticate with Shopify through basic HTTP authentication. This can be accomplished by completing the Authentication properties like below: CallRESTEndpointFNC Step 3: Querying And Modifying Data The below examples are demonstrations of interacting with the Shopify API and Linx. Querying Customers The below example deals with the object. The function makes a request to the endpoint which returns a JSON string response containing a list of all the customers. The response is then imported by Linx into a custom type object which is then used to structure the response body. Customer GET /customers First, create a new and give it the name of . Drag a onto the canvas. Function GetCustomers CallRESTEndpointFNC GetCustomers Complete the Basic authentication details by referencing the API Key and password from earlier. Then configure the URL to be like the below (using the expression editor): = $.Settings.shopify_baseURI + + "/customers" ".json" Right-click on the and and , this will expose the runtime values of the objects in scope. Next, debug the function and take note of the response body returned in the Debug Values panel. The response body is a JSON string like below: CallRESTEndpointFNC Add breakpoint Enable logging GetCustomers { : [
    { : , : , : , : , : , : , : , : , : , : , : , : , : , : , : , : , : , : , : , : [
        { : , : , : , : , : , : , : , : , : , : , : , : , : , : , : , : , : }
      ], : , : , : [], : , : { : , : , : , : , : , : , : , : , : , : , : , : , : , : , : , : , : }
    }
  ]
} "customers" "id" 207119551 "email" "bob.norman@hostmail.com" "accepts_marketing" false "created_at" "2021-01-01T14:46:48-05:00" "updated_at" "2021-01-01T14:46:48-05:00" "first_name" "Bob" "last_name" "Norman" "orders_count" 1 "state" "disabled" "total_spent" "199.65" "last_order_id" 450789469 "note" null "verified_email" true "multipass_identifier" null "tax_exempt" false "phone" "+16136120707" "tags" "" "last_order_name" "#1001" "currency" "USD" "addresses" "id" 207119551 "customer_id" 207119551 "first_name" null "last_name" null "company" null "address1" "Chestnut Street 92" "address2" "" "city" "Louisville" "province" "Kentucky" "country" "United States" "zip" "40202" "phone" "555-625-1199" "name" "" "province_code" "KY" "country_code" "US" "country_name" "United States" "default" true "accepts_marketing_updated_at" "2005-06-12T11:57:11-04:00" "marketing_opt_in_level" null "tax_exemptions" "admin_graphql_api_id" "gid://shopify/Customer/207119551" "default_address" "id" 207119551 "customer_id" 207119551 "first_name" null "last_name" null "company" null "address1" "Chestnut Street 92" "address2" "" "city" "Louisville" "province" "Kentucky" "country" "United States" "zip" "40202" "phone" "555-625-1199" "name" "" "province_code" "KY" "country_code" "US" "country_name" "United States" "default" true This object contains a parent object containing a “customers” child object which contains a list of objects which contain the fields related to the product we are creating. Next we need to create a custom data type so Linx will know what to parse the response body into, this will allow us to work with individual field values in subsequent operations. Copy the value of from the Debug values panel or from the . Import the copied JSON string as a new . This will create a data object or type for you to structure the response data into. Configure the to have an of the newly imported user-defined type. You can then return the details as an output of the custom function. CallRESTEndpoint.ResponseBody API documentation Type CallRESTEndpointFNC Output type Adding Products In the below example, a will be added to Shopify. This will involve making a request to the A JSON structure containing details of a will be submitted as the request body. Product POST /products endpoint. Product Example request: POST /admin/api/ /products.json
{ : { : , : , : , : , : [ , , ]
  }
} 2021 -01 "product" "title" "Burton Custom Freestyle 151" "body_html" "<strong>Good snowboard!</strong>" "vendor" "Burton" "product_type" "Snowboard" "tags" "Barnes & Noble" "John's Fav" "Big Air" In the below example we are going to create a user-defined function that will take in the details of the product as the input parameters. These details will then be assigned to the user-defined type and submitted in a request. First, In order to submit a valid JSON data structured like the above, you need to create a user-defined that matches the fields in the product object, either manually or by importing the provide JSON example as a new Type with the name of “newProduct”. This will create two types, a parent newProduct that holds a child newProduct_product. Type This user-defined type is then available to reference across your Linx application to structure data. Next, we are going to create a user-defined function that will take in data in this format and then submit this data to the Shopify API. To do this, create a new user-defined function with the name of “CreateProduct”, with an input parameter with the name of the product and the Type of product. When this CreateProduct user-defined function is called from somewhere else in the application, the structure of the child object will be able to be used to pass data through. The parent product is not needed as an input parameter as it just creates another level, we can assign it later inside the function. For the request body, we are going to submit a newProduct type from earlier in a JSON format. Currently, the CreateProduct function takes in an input parameter of the type newProduct_product. However, for the request to succeed, we need to submit the entire structure of newProduct. To do this, we need to assign the input parameter productDetails details as the child newProduct_product of newProduct. To do this, drag an instance of the newProduct type from the Solution Explorer onto the CreateProduct canvas. Expand its field values properties and in the product field, reference the input parameter productDetails. Now to make the request using the newProduct type. Inside the CreateProduct function, drag a onto the canvas. Configure the authentication credentials and the request URL. CallRESTEndpointFNC Now we need to submit the local instance of newProduct as the request body. Change the of the request to POST and set the as JSON. For the property, reference the local newProduct. Method Body format Body Now we can make a request to test it out. A new product should be visible on the Shopify products dashboard: You can then import the response as a new user-defined type and set it as the of the CallRESTEndpointFNC. When a new product is now created, the newly added details and some metadata including the new id are then available in subsequent functions such as logging them to a database with an or writing to a file using a . Output type ExecuteSQLFNC TextFileWriteFNC Previously published at https://community.linx.software/community/t/shopify-integration/506

How to Build a Shopify Integration

A step by step tutorial to analyse sentiment of Amazon product reviews with the FastText API This blog provides a detailed step-by-step tutorial to use FastText for the purpose of text classification. For this purpose, we choose to perform sentiment analysis of customer reviews on Amazon.com and also elaborate on how the reviews of a particular product can be scraped for performing sentiment analysis on them hands on, the results of which may be analysed to decide the quality of a product based on the given feedback, before purchase. What is FastText? Text classification has become an essential component of the commercial world; whether it is used in spam filtering or in analysing sentiments of tweet sor customer reviews for E-Commerce websites, which are perhaps the most ubiquitous examples. FastText is an open-source library developed by the Facebook AI Research (FAIR), exclusively dedicated to the purpose of simplifying text classification. FastText is capable of training with millions of example text data in hardly ten minutes over a multi-core CPU and perform prediction on raw unseen text among more than 300,000 categories in less than five minutes using the trained model. Pre-Labelled Dataset for Training A manually annotated dataset of amazon reviews obtained from Kaggle.com containing few million reviews was collected and used for training the model after conversion to FastText format. The data format for FastText is as follows: __label__ __label__ ... < > X < > Y < > Text where X and Y represent the class labels. In the dataset we use, we have the review title prepended to the review, separated by a ‘ ’ and a space. : A sample from the training data file is given below, the datasets for training and testing the models can be found here in the website. Kaggle.com __label__2 Great CD: My lovely Pat has one the GREAT voices her generation. 
I have listened this CD YEARS I
still LOVE IT. I makes feel better. A bad mood just
evaporates sugar the rain. This CD
just oozes LIFE. Vocals are jusat STUUNNING lyrics just kill. One life gems. This a desert isle CD my book.
Why she never made it big just beyond .
everytime I play this, no matter black,
white, young, old, male, female EVERYBODY
says one thing of of to for and When 'm in a good mood it me like in and of 's hidden is in is me "Who was that singing ?" Here, we have only two classes 1 and 2, where signifies that the reviewer gave either 1 or 2 stars for the product, while indicates a 4 or 5 star rating. __label__1 __label__2 Training FastText for Text Classification Pre-process and Clean Data Execute the following command to generate a preprocessed and cleaned training data file after normalizing text case and removing unwanted characters. <path training > | sed - “s/$[.\!?,’/()]$/ \ /g” | “[:upper:]”
“[:lower:]” > 
<path -processed output > cat to file e 1 tr to pre file Setup FastText Let us start by downloading the : most recent release $ wget https://github.com/facebookresearch/fastText/archive/v0.1.0.zip $ unzip v0.1.0.zip Move to the fastText directory and build it: $ fastText-0.1.0 cd $ make Running the binary without any argument will print the high level documentation, showing the different use cases supported by fastText: >> ./fasttext
usage: fasttext < > < >
The commands supported fasttext are:
  supervised              train supervised classifier
  quantize                quantize model reduce memory usage
  test                    evaluate supervised classifier
  predict                 predict most likely labels
  predict-prob            predict most likely labels probabilities
  skipgram                train skipgram model
  cbow                    train cbow model
  print- -vectors      print vectors given trained model
  print- -vectors  print vectors given trained model
  nn                      query nearest neighbors
  analogies               query analogies command args by a a to the a with a a word word a sentence sentence a for for In this tutorial, we mainly use the , and subcommands, which corresponds to learning (and using) text classifier. supervised test predict Training the model The following command is used to train a model for text classification: ./fasttext supervised - < pre-processed training file> -output
< save model> -label __label__ input path to path to The command line option refers to the training file, while the option refers to the location where the model is to be saved. After training is complete, a file , containing the trained classifier, is created in the given location. -input -output model.bin Optional parameters for improving models Increasing number of epochs for training By default, the model is trained on each example for 5 epochs, to increase this parameter for better training, we can specify the -epoch argument. Example: ./fasttext supervised - < pre-processed training file> -output 
< save model> -label __label__ -epoch input path to path to 50 Specify learning rate Changing learning rate implies changing the learning speed of our model is to increase (or decrease) the learning rate of the algorithm. This corresponds to how much the model changes after processing each example. A learning rate of 0 would means that the model does not change at all, and thus, does not learn anything. Good values of the learning rate are in the range . 0.1 - 1.0 The default value of lr is 0.1. Here’s how we specify this parameter. ./fasttext supervised - <path -processed training > -output
 <path save model> -label __label__ - input to pre file to lr 0.5 Using n-grams as features This is a useful step for problems depending on word order, especially sentiment analysis. It is to specify the usage of the concatenation of consecutive tokens in a n-sized window as features for training. We specify parameter for this (ideally value between 2 to 5): -wordNgrams ./fasttext supervised - < pre-processed training file> -output 
< save model> -label __label__ -wordNgrams input path to path to 3 Test and Evaluate the Model The following command is to test the model on a pre-annotated test dataset and compare the original labels with the predicted labels of each review and generate evaluation scores in the form of precision and recall values. The precision is the number of correct labels among the labels predicted by fastText. The recall is the number of labels that successfully were predicted. ./fasttext test < model> < test file> k path to path to where the parameter represents that the model is to predict the top labels for each review. k k The results obtained on evaluating our trained model on a test data of 400000 reviews are as follows . As observed, a precision, recall of 91% is obtained and the model is trained in a very quick time. N Number of examples: 400000 P@ 1 0.913 R@ 1 0.913 400000 Analyse Sentiments of Real-Time Customer Reviews of Products on Amazon.com Scrape Amazon Customer Reviews We use an existing python library to scrape reviews from pages. To setup the module, In your command prompt/terminal type: pip amazon-review-scraper install Here’s a sample code to scrape review of a particular product, given the url of the web page: amazon_review_scraper amazon_review_scraper

url = input( )
start_page = input( )
end_page = input( )
time_upper_limit = input( )
file_name = scraper = amazon_review_scraper.amazon_review_scraper(url, start_page, end_page, time_upper_limit)
scraper.scrape()
scraper.write_csv(file_name) from import "Enter URL: " "Enter Start Page: " "Enter End Page: " "Enter upper limit of time range (Example: Entering the value 5 would mean the program will wait anywhere from 0 to 5 seconds before scraping a page. If you don't want the program to wait, enter 0): " "amazon_product_review" : While entering the URL of the customer review page of a particular product, ensure that you append if it does not exist already, for the scraper to function properly. NOTE &pageNumber=1 The above code scrapes the reviews from the given url and creates an output csv file in the following format: ( Dataset Link ) From the above csv file, we extract the Title and the Body and append them together separated by a and a space as in the training file, and store them in a separate txt file for prediction of sentiments. ‘: Prediction of Sentiments of Scraped Data ./fasttext predict < model> 
< test file> 
k > < prediction file> path to path to path to where signifies that the model will predict the top labels for each review. k k The labels predicted for the above reviews are as follows: 2 1 2 2 2 2 2 2 1 2 2 __label__ __label__ __label__ __label__ __label__ __label__ __label__ __label__ __label__ __label__ __label__ Which are quite accurate as verified manually. The prediction file can then be used for further detailed analysis and visualization purposes. Thus, in this blog, we learnt using the FastText API for text classification, scraping Amazon Customer Reviews for a Given Product and predicting their sentiments with the trained model for analysis. If you have any queries or suggestions, I would love to hear about it. Please write to me at abhishek.narayanan@dataturks.com.

Text Classification Simplified with Facebook’s FastText

About the Authors: This was published by a group of researchers from (Facebook AI research). The original authors are , , and . paper FAIR Piotr Bojanowski Edouard Grave Armand Joulin Tomas Mikolov The for this paper is available . ready-to-run code here on Google Colab The Basic Idea behind Word Vectors: For most of the Natural Language Processing related tasks like text classification, text summarization, text generation etc, we need to perform various computations in order to achieve maximum precision on these tasks. In order to perform these computations, we need a for various components of Language like words, sentences and syllables. numerical based representation We assign multi-dimensional vectors to represent the words in the language to get a vector-based representation of the language. IMG SRC : deeplearning.ai by Andrew Ng Each dimension of such vector can capture the different characteristics and features of the words. Word vectors are also referred to as word embeddings in the Natural Language scientific community. You can know more about word vectors . here Earlier Work: For the last two decades, many to calculate word vectors were introduced. Many of them used computationally expensive models to calculate the word embeddings. approaches In 2013, Tomas Mikolov et al introduced a simpler and efficient way of calculating word embeddings using a computationally cheaper model: . word2vec with skipgram or CBOW approach This paper is an extension of Mikolov’s word2vec SkipGram model. The Basic idea behind Word Vectors with Subword Information: Mikolov’s word2vec model provided an excellent set of word embeddings for most of the large publicly available datasets. The one huge of word2vec model was that and only assigned the features based on the semantic context of the word. The major source of learning for Mikolov’s word2vec model was the external neighbors of the word. This also . limitation it ignored the morphological structure of the words limited it to learn the word vectors for only the words present in the vocabulary So, in languages like where the internal morphological structures of the words hold certain importance, the word2vec model failed to capture all the features of the available text data. Turkish, German and Czech This limitation has been addressed in this paper by using to cover the information from morphological structures of the words. subword information GOALS: Our Goals can be listed as follows : For a of ‘ , we need to learn a vector representation for every ‘ vocabulary size W’ word w’. The word vector representation should consider the context of the surrounding words. The word vector representation should also consider the internal morphological structure of the word. APPROACH: The first thing to be done here is setting up a word dictionary with an index for every word. Next, we define a window size which will give us the context words for every target word. For example, a for the target word “ in the sentence: “ ” , the context words would be: window size of 2 army ” I have an army. We have a hulk ( “have”, “an”, “we”, “have” ) The next step is to calculate the score for context between words. Let's assume a function to numerically score the context between the < and the < . Where < and < are the vector representations for these words and < represents the . S(w,c) → R target word w> context word c> w> c> R> single dimension real numbers space Further, we can define the probability of < occurring as a context word as a softmax function as follows : c> Here <Wc> and <Wt> are the vectors representing the target and context words and W is the vocabulary size. But this probability distribution only considers the probability of occurrence of a single context word to that of the whole vocabulary. Thus this cannot be used to define our to train the model. Predicting context words can be seen as a set of binary classification problems. Therefore, we use the binary logistic loss to get the following as the : objective function negative log-likelihood objective function The final objective function for training. Now, to parameterize the model, we the as the between the vectors and . define context score S(w,c) scalar product <w> <c> Until here, the approach is exactly the same as the one given by Mikolov. Clearly, the current approach only considers the neighboring context words for calculating the features and completely ignores the structure of the word itself. The Subword Model: To address this issue, we represent the word “ as in the word. The : . This helps in distinguishing the suffixes and prefixes from the rest of the character sequences. We also add the complete word as a sequence into this bag of n-grams. w” a bag of all possible character n-grams word is padded by a set of unique symbols like the angled brackets <WORD> For n = 3, the word “ ” will give us the following n-grams bag represented as EXAMPLE: where Gw = [ <wh, whe, her, ere, re>, <where> ] Gw Now, every character sequence in the n-grams bag is denoted by a vector and the word vector is given by the sum of the vectors of all the n-grams in that word. We also change the context score function to : <z> <w> This allows the sharing of representations across words, . This way, the subword information is utilized in the learning process of calculating word embeddings. thus allowing to learn reliable representation for rare words This model can be memory-wise costlier. So, the authors use Fowler-Noll-Vo hashing function ( ) to hash the character sequences. Ultimately, a word is represented by its index in the word dictionary and the set of hashed n-grams it contains. FNV-1a variant Optimization: Given, the as the before, we can optimize to it ( and hence ) by an optimization method. The authors have used with for all their experiments. The same optimization method has been used by Mikolov et al in their word2vec model. negative log-likelihood objective function minimize maximize the likelihood Stochastic Gradient Descent Linear Learning Rate decay Implementation Details: The authors have tried to maintain maximum similarity between their approach and Mikolov et al’s approach. = Word vector size 300 = Negative Sampling Size 5 samples per positive sample = . Word rejection criteria If the word occurs less than times in the corpus, it is removed from the dictionary This implementation is than Mikolov et al’s SkipGram implementation. 1.5x slower EXPERIMENTS: Datasets: Wikipedia dumps in nine languages: Arabic, Czech, German, English, Spanish, French, Italian, Romanian and Russian. Human similarity judgment: C omputing Spearman’s rank correlation coefficient between human judgment and the cosine similarity between the vector representations. Word analogy tasks: This is tightly related to the choice of the length of character n-grams that we consider. The analogy accuracy is split as Semantic (meaning-based) and Syntactic (syntax and grammar-based). Other Experiments: Further the authors also experiment with Language Modelling tasks, n-grams size variation, morphemes and qualitative analysis. CONCLUSION: Some very interesting results can be concluded, which will completely justify the inclusion of subword information for generating word vectors. Effect of Training Size: Since we exploit character-level similarities between words, we are able to better model infrequent words. Therefore, we should also be more robust to the size of the training data that we use. Effect of Training size on the performance of the model. SISG performs better with smaller training data size. Word similarity for OOV words: Our model is capable of building word vectors for words that do not appear in the training set. For such words, we simply average out the vector representation of its n-grams. Out of Vocabulary words as queries. Some of the interesting out of the vocabulary vector predictions can be explained as follows : & ‘ ‘rarity’ scarceness’ : ‘ scarce’ roughly matches ‘rarity’ while the suffix ‘-ness’ matches ‘-ity’ very well. & ‘ . ‘preadolescent’ young’: due to the subwords ‘-adolescent’ and ‘pre-’ , semantic similarity is obtained with the word ‘young’ REFERENCES: Enriching word vectors with Subword Information | ACL 2017 Original Paper Mikolov et al word2vec | Original Paper

Enriching Word Vectors with Subword Information [PAPER SUMMARY]

Whether you are new to deep learning or a seasoned veteran, setting up an environment for training a neural network can be painful sometimes. Would it be awesome to make training a neural network as simple as loading a webpage followed by a few clicks and you are ready to make inference with it right away? In this tutorial, I will show you how to build a model with the on-browser framework TensorFlow.js with data collected from your webcam and train on your browser. To make the model useful, we will turn a webcam into a controller for the legendary game — Pong. Let’s play the game first Instruction to get the web application served locally on your computer, Download the and extract it to your local machine. dist.zip Install an HTTP server, my recommendation is to install globally by npm, http-server npm install -g http-server You ask what is npm? It is a package installer for Node.js like pip for Python and can be acquired . here Run the following command in a command line where the folder is located to serve the web app on your local machine at a port, say 1234. dist http-server dist --cors -p 1234 -s Point a browser window to , I have tested on Chrome and Firefox. http://localhost:1234 When the page finishes loading, start by collect training images for three moves, left, middle and right. One tip here, balance the training samples, maybe around 20 samples for each case. Click the “TRAIN”, and it will start the training process with the loss shows. The loss is not changing means the training is over, now click “PLAY” to start the game. If you want to start over, click “RESET”. After you get tired of beating the computer with head or hand with your choice, let’s have a look at how the game is built. There are two models used in this tutorial, the first one is a pre-trained convolutional network exported from Keras, it is responsible for extracting image features from webcam images. The second model builds and trains on your browser to make predictions for game control with the image features. It is a regression model predicts values range between -1~1 to control the speed of the player’s paddle. It is essentially a transfer learning task. More on the topic of transfer learning, refer to my previous series — . Without further due, let’s dive in. You can download the source code now from my GitHub repo . Gentle guide to setup Keras deep learning framework and build a travel recommendation engine webcam-pong Export pre-trained model to tfjs You can skip this section if you just want to learn the web application part. Let’s start by export a pre-trained convolutional network to TensorFlow.js(tfjs) format. I pick trained with ImageNet datasets in this tutorial but you can go with other models like MobileNet. Try to avoid large deep convolutional networks like the ResNets and VGGs even though they might provide slightly higher accuracy but not fit edge devices like our case running on a browser. DenseNet The first step is to save the pre-trained DenseNet Keras model to a file in Python script. .h5 Then we run the conversion script to convert the .h5 file to tfjs files optimized for browser caching. Before continuing, install the tensorflowjs conversion script python package through pip3. pip3 install tensorflowjs We can now generate tfjs files by running, cd ./tfjs-densenettensorflowjs_converter --input_format keras ./model.h5 ./model You will see a folder named with several files inside. The file defines the model structure and the path to weights files. And the pre-trained model is ready to be served for the web app. For example, you can rename the folder to and copy to your web app served folder, then the model can be loaded like this. model model.json model serveDenseNet is the web app URL or if you serve it locally on port 1234 it will be . The statement simply allows the web app to load the model in the background without freezing the main user interface. window.location.origin localhost:1234 await Also, realize that since the model we loaded is an image classification model with layers at the top we don’t need, we only want the feature extraction part of the model. The solution is to locate the topmost convolutional layer and truncate the model shown in the previous code snippet. Generate training data from a webcam To prepare the training data for the regression model, we will grab some images from a webcam and extract their feature with the pre-trained model within the web app. To simplify the user interface for acquiring training data, we only label images with one of the three values [-1, 0, 1]. For each image acquired through the webcam, it will be feed into the pre-trained DenseNet to extract features and saved as a training sample. After passing an image through the feature extractor model, a 224 by 224 color image will have its dimension reduced to image features tensor of shape [7 ,7, 1024]. The shape depends on the pre-trained model you choose and can be obtained by calling on the layer we picked in the previous section like this. outputShape modelLayerShape = layer.outputShape.slice(1) The reason to use the extracted image features as training data instead of the raw images is two folds. First it saves memory to store the training data, second, it reduces the training time by not running the feature extraction model. This following snippet shows how an image is captured by a webcam, has its features extracted and aggregated. Note that all image features are saved in the form of tensors which means if your model runs with the browser’s WebGL backend, there is a limit on how many training samples it can safely contain at a time in GPU memory. So don’t expect training your model with thousands or even hundreds of image samples depends on your hardware. Build and train the neural network Build and train your neural network without uploading to any cloud service protects your privacy since the data never leave your device and watch it happen on your browser make it even cooler. The regression model takes image features as input flatten it to a vector then followed by two fully connected layers and generates one floating number to control the game. The last fully connected layer takes no activation function since we want it to produce real numbers between -1 to 1. The loss function we pick is mean squared error during the training to minimize the loss. More on the choice, read my post — . How to choose Last-layer activation and loss function The following code will build, compile and fit the model. Looks quite similar to Keras’ workflow right? Turning webcam into a Pong controller As you might expect to predict with an image resembles Keras syntax as well. The image is first converted to image features then pass on to the trained regression neural network which outputs a controller value between -1 to 1. Once you got the model trained and the game is up and running, the predicted value passes down to control the speed of player’s paddle going left or right with variable speed through this call . You can start and stop the game by calling, pong.updatePlayerSpeed(value) it happens if you press the button pong.startGameplay() Play will be called if you click the button. pong.stopGameplay() Reset The aggressiveness of the paddle movement can be adjusted by calling , the current multiplier value is set to 12 in the Pong's class constructor. pong.updateMultiplier(multiplier) Conclusion and further thought In this tutorial, you have learned how to train a neural network on a browser with TensorFlow.js and turn your webcam into a Pong controller recognizing your moves. Feel free to check out my source code and experiment with it, modified it and see how it turns out, like the activation functions, loss function and swap to another pre-trained model, etc. The beauty of training a neural network on a browser with instant feedback enables us trying new ideas and get results faster for our prototypes also make it more readily accessible to the general public. Check out the full source on my GitHub repo . webcam-pong _webcam-pong - Transfer Learning to play Pong via the Webcam_github.com Tony607/webcam-pong Originally published at www.dlology.com .

How to train neural network on browser

Word2Vec; the Steroids for Natural Language Processing Let’s start with the Basics. Word Vectors Q) What are word vectors? Representation of words with numbers. Ans) Q) Why Word Vectors? I’ll sum it up with three main reasons: Ans) 1. Computer cannot do computations on strings. 2. Strings don’t hold much information themselves. explicit 3. Words Vectors are usually dense vector representations. Q) So what is Explicit information? Yes, the word itself doesn’t say much about what it represents in real life. Example: Ans) The string “cat” just tells us it has three alphabets “c”, ”a” and “t”. It has no about the animal it represents or the count or the context in which it is being used. information Q) Dense Vector Representation? Short answer (for now), these vectors can hold Enormous information compared to their size. Ans) Q) Types of Word Vectors? A) There are two main categories: : Use stats to compute probability of a word co-occurring with respect to it’s neighbouring words. Frequency based : Use predictive analysis to make a weighted guess of a word co-occurring with respect to it’s neighbouring words. Prediction based Predictions can be of two types: : Try to guess a word w.r.t. context of text Semantic : Try to guess a word w.r.t. syntax of text Syntactic Q) Difference between syntax and context based vectors? Let’s look at an example: Ans) Consider the following sentence “Newton does not like apples.” Semantic Vectors are concerned with ‘ ’. In this case “Newton” and “Apples” Who or What are the entities, a text is about? Syntactic Vectors are concerned with ‘ ’. In this case “Not” and “Like” What about those entities is being said? Word2Vec Is one of the most widely used form of word vector representation. First coined by Google in Mikolov et el . It has two variants: : This model tries to predict a word on bases of it’s neighbours. CBOW (Continuous Bag of Words) 2. : This models tries to predict the neighbours of a word. SkipGram Statistically it has the effect that CBOW smoothes over a lot of the distributional information (by treating an entire context as one observation). For the most part, this turns out to be a useful thing for smaller datasets. However, skip-gram treats each context-target pair as a new observation, and this tends to do better when we have larger datasets. - Tensorflow In simpler words, tends to find the probability of a word occurring in a neighbourhood (context). So it . CBOW generalises over all the different contexts in which a word can be used Whereas tends to . So SkipGram needs enough data w.r.t. each context. Hence requires more data to train, also SkipGram (given enough data) contains more knowledge about the context. SkipGram learn the different contexts separately SkipGram : These techniques do not need tagged dataset (though a tagged dataset can be used to include additional information as we’ll see later). So any large text corpus is effectively a dataset. As the tag to be predicted are the words already present in the text. NOTE We will focus on SkipGram as large enough datasets (Wikipedia, Reddit, Stackoverflow etc.) are available for download. SkipGram First we decide what context are we looking for in terms of what will be our (to be predicted), (on bases of which we predict) and are we looking for context (size of window). target words source words how far Example: Considering Window size to be 3 Type 1 Considering the middle word as the source word. The next and previous words as the target words. fig no. 1 fig no. 2 Type 2 Considering the first word as the source word. The following two words as the target words. fig no. 3 fig no. 4 In both the types, source word is surrounded by words which are relevant to a context of that source word. Like ‘Messi’ will usually be surrounded by words related to ‘Football’. So after seeing a few examples, word vector of ‘Messi’ will start incorporating context related to ‘Football’, ‘Goals’, ‘Matches’ etc. In case of ‘Apple’, its word vector would do the same but for both the company and the fruit (see fig no. 6). fig no. 5 Word2Vec’s Neural Network W1(s) and W2(s) contain information about the words. The information in W1 and W2 are combined/averaged to obtain the Word2Vec representations. Say the Size of W(s) was 400, the Word2Vec representation of ‘apple’ would look something like array([-2.56660223e-01,  -7.96796158e-02,  -2.04517767e-02, -7.34366626e-02,   3.26843783e-02,  -1.90244913e-02, 7.93217495e-02,    4.07200940e-02,  -1.74737453e-01, ..... 1.86899990e-01,    -4.33036387e-02,  -2.66942739e-01, -1.00671440e-01],   dtype=float32) Now a simple sentence like “Apple fell on Newton” containing 4 words, with help of Word2Vec, can be converted into 4*400 (1600) numbers; . So now we also know that the text is talking about a person, science, fruit etc. each [1] containing explicit information [1] : hence Dense Vector representation Visualisation Visualising Word2Vec directly is currently impossible for mankind (because of high dimensionality like 400). Instead we use techniques like , , etc. dimensionality reduction multidimensional scaling sammon’s mapping nearest neighbor graph The most widely algorithm is (t-SNE). Christopher Olah has an amazing about Dimensionality Reduction. t-Distributed Stochastic Neighbour Embedding blog The end result of t-SNE on Word2Vec looks something like fig no. 6 Multiple contexts of Apple This figure shows that Apple lies between Companies (IBM, Microsoft) and Fruits (Mango). That’s because the Word2Vec representation of Apple contains information about both the Company Apple and the Fruit Apple. Distance Between Apple and Mango : 0.505 Apple and IBM : 0.554 Mango and IBM : 0.902 And fig no. 7 Combining contexts of multiple words This figure shows that by combining directions of two vectors ‘State’ and ‘America’, the resultant vector ‘Dakota’ is relevant to original vectors. So effectively State + America = Dakota State + Germany = Bavaria Other examples are : German + Airlines = Lufthansa King + Woman — Man = Queen Implementations Gensim and Tensorflow, both have pretty impressive implementations of Word2Vec. This is excellent of ’s implementation and has a . blog Gensim Tensorflow tutorial Issues By default, Word2Vec model has one representation per word. A vector can try to accumulate all contexts but that just ends up generalising all the contexts to at least some extent, hence precision of each context is compromised. This is especially a problem for words which have very different contexts. This might lead to one context, over powering others. Like : There will be only one Word2Vec representation for ‘apple’ the company and ‘apple’ the fruit. Example 1: ‘maiden’ can be used for a woman, a band (Iron Maiden), in sports etc. When you try to find most similar words to ‘maiden’ [(u'odi_debut', 0.43079674243927),(u'racecourse_debut', 0.42960068583488464),.....(u'marathon_debut', 0.40903717279434204),(u'one_day_debut', 0.40729495882987976),(u'test_match_debut', 0.4013477563858032)] It is clearly visible that the context related to ‘sports’ has overpowered others. Even combining ‘iron’ and ‘maiden’, doesn’t resolve the issue as now context of ‘iron’ overpowers. [(u'steel', 0.5581518411636353),(u'copper', 0.5266575217247009),.....(u'bar_iron', 0.49549400806427)] Example 2 A word could be used as a verb and a noun but with completely different meanings. Like the word ‘iron’. As a verb it is usually used to smooth things with an electric iron but at a noun it is mostly used to denote the metal. When we find the nearest neighbours of ‘iron’ [(u'steel', 0.5581518411636353),(u'copper', 0.5266575217247009),.....(u'bar_iron', 0.49549400806427)] There is negligible reference to verb counterpart. Variants Variant 1 : Compound Noun based Word2Vec By replacing nouns (like ‘iron’ and ‘maiden’) by compound nouns (like ‘iron_maiden’) in training set. “Iron Maiden is an amazing band” becomes “ Iron_Maiden is an amazing band” So the context of the compound noun stands out and is remarkably accurate! Result: Most relevant words w.r.t ‘iron_maiden’ are: [(u'judas_priest', 0.8176089525222778),(u'black_sabbath', 0.7859792709350586),(u'megadeth', 0.7748109102249146),(u'metallica', 0.7701393961906433),..... That’s hardcore, literally! Here is a Python code for converting nouns into compound nouns ( ), in order to create the training set for training Word2Vec. with adj-noun pairing as well fig no. 8 This Python code is for converting nouns into compound ( ). only noun-noun paring fig no. 9 Variant 2 : Sense2Vec (Note : Non NER implementation of Sense2Vec) Taking the above mentioned variant one step further by adding Part Of Speech (P.O.S) tags to the training set. Example: “I iron my shirt with class” becomes “I /PRP iron /VBP my /PRP$ shirt /NN with /IN class /NN . /. ” Or “I**/NOUN** iron**/VERB** my**/ADJ** shirt**/NOUN** with**/ADP** class**/NOUN** . ” /PUNCT Result: Now most relevant words w.r.t ‘iron/VERB’ are: [(u'ironing/VERB', 0.818801760673523),(u'polish/VERB', 0.794084906578064),(u'smooth/VERB', 0.7590495347976685),..... (Refer ‘Example 2’ of ‘Issues’ section for comparison) Below is visualisation of Sense2Vec fig no. 10 Below is the Python code for preparing training dataset for Sense2Vec fig no. 11 These codes are available at . github Conclusion Word2Vec can hold enormous information compared to their size! They can learn both semantics and syntax The one problem is generalisation over multiple contexts but that too can be tackled with additional modification of training text They are computation friendly as they are all array of numbers The relationship between vectors can be discovered with just linear algebra Word2Vec (Part 2) Use Cases Next : Prev : Natural Language Processing (NLP)

Stories, News & Mentions about SHOPGRAM

Shopgram’s Stories on HackerNoon