ThoughtSpot is disrupting the BI industry with search-driven analytics for the enterprise

ThoughtSpot

thoughtspot

A journey from the origins of computing and data analytics to what we now call the "Modern Data Stack". What comes next? The Origins of Computing and Data Analytics The origins of computing and began in the mid-1950s and started taking shape with the introduction of SQL in 1970: data analytics 1954: - “ ”, machine translation of Russian to English. Natural Language Processing (NLP) Georgetown-IBM experiment 1960: Punch Cards 1970: Structured Query Language (SQL) 1970s: - Create a language to “allow executives to build models without intermediaries” Interactive Financial Planning Systems 1972: - One of the earliest applications of modern computing, a natural language information retrieval system, helped geologists access, compare and evaluate chemical-analysis data on moon rock and soil composition C, LUNAR 1975: - The first Online Analytical Processing (OLAP) system, intended to analyse business data from different points of view Express 1979: - The first spreadsheet computer program VisiCalc 1980s: - “Computerized Collaborative Work System” Group Decision Support Systems The “Modern Data Stack” The "Modern Data Stack" is a set of technologies and tools used to collect, store, process, analyze, and visualize data in a well-integrated cloud-based platform. Although QlikView was pre-cloud, it is the earliest example of what most would recognize as an analytics dashboard used by modern platforms like Tableau and PowerBI: 1994: - “Dashboard-driven Analytics” QlikView 2003: Tableau 2009: - “Computational Search Engine” Wolfram Alpha 2015: PowerBI 2017: - “Search-driven Analytics” ThoughtSpot Paper, Query Languages, Spreadsheets, Dashboards, Search, what next? Some of the most innovative analytics applications, at least in terms of user experience, convert human language to some computational output: A tale as old as time, LUNAR was first developed in the 70s to help geologists access, compare, and evaluate chemical analysis data using natural language. Salesforce WikiSQL introduced the first extensive compendium of data built for the text-to-SQL use case but only contained simple SQL queries. The Yale Spider dataset introduced a benchmark for more complex queries, and most recently, BIRD introduced real-world “dirty” queries and efficiency scores to create a proper benchmark for text-to-SQL applications. Text-to-SQL: Wolfram Alpha, ThoughtSpot Text-to-Computational-Language: ChatGPT Advanced Data Analysis Text-to-Code: Is "Conversation-Driven Data Analytics" a natural evolution? UX of modern analytics interfaces like , becoming more intuitive, enabled by NLP and LLMs search and chat are evolving Analytics interfaces have origins in but decision-makers are still largely enabling decision-makers, reliant on data analysts Many decision-maker , best suited to “throwaway analytics” queries are ad-hoc Insight generation is a where many insights are gained in conversations about data, possibly with peers creative process The data analytics , from the imagination of analysis to the presentation of results workflow is disjointed Acknowledgments Dates for the section "The Origins of Computing and Data Analytics" thanks to and . https://web.paristech.com/hs-fs/file-2487731396.pdf http://dssresources.com/history/dsshistoryv28.html Also published here.

A Brief History of Computing and Data Analytics - From Punch Cards to the "Modern Data Stack"

কম্পিউটিং এবং ডেটা বিশ্লেষণের সংক্ষিপ্ত ইতিহাস - পাঞ্চ কার্ড থেকে "আধুনিক ডেটা স্ট্যাক" পর্যন্ত

Eine kurze Geschichte der Informatik und Datenanalyse – von Lochkarten bis zum „modernen Datenstapel“

Una breve historia de la informática y el análisis de datos: desde las tarjetas perforadas hasta la "pila de datos moderna"

Une brève histoire de l'informatique et de l'analyse de données - Des cartes perforées à la « pile de données moderne »

कंप्यूटिंग और डेटा एनालिटिक्स का संक्षिप्त इतिहास - पंच कार्ड से "आधुनिक डेटा स्टैक" तक

コンピューティングとデータ分析の歴史 - パンチカードから「最新のデータスタック」まで

컴퓨팅 및 데이터 분석의 간략한 역사 - 펀치 카드에서 "현대 데이터 스택"까지

Uma Breve História da Computação e Análise de Dados – Dos Cartões Perfurados à “Pilha de Dados Moderna”

Краткая история вычислений и анализа данных - от перфокарт до «современного стека данных»

Bilgi İşlem ve Veri Analitiğinin Kısa Tarihi - Delikli Kartlardan "Modern Veri Yığınına"

Sơ lược về lịch sử tính toán và phân tích dữ liệu - Từ thẻ đục lỗ đến "Ngăn xếp dữ liệu hiện đại"

计算和数据分析简史 - 从打孔卡到“现代数据堆栈”

Welcome to HackerNoon’s interview series. This section is curated by our Editorial team to spotlight bold and disruptive startups across the globe. If you’ve been nominated, and answer these questions Startups-Of-The-Year create a brand profile here. Hey Hackers, BlueCloud has been nominated in HackerNoon's annual Startup of the Year awards in Tampa, United States. Please vote for us . here Read more about us below to understand why we deserve your vote. Meet BlueCloud BlueCloud is a global digital transformation company delivering best-of-breed cloud data and analytics services to partner clients. Our company takes a data-driven approach to digital transformation and consulting services, implementing modern cloud data and analytics solutions for our mid-market and Fortune 500 clients by leveraging premium yet cost-effective delivery models, next-generation technologies, and a world-class team of IT leaders. My Role Kerem Koca – BlueCloud CEO & Co-Founder As BlueCloud’s CEO and Co-Founder, I build high-performing teams and deliver superior IT solutions that drive sustainable growth for organizations. I am an industry pioneer in digital transformation and cloud-native data analytics implementation and consulting services, co-founding BlueCloud (formerly Bluenet) in 2004 and delivering digital transformation strategies to enterprises for the past two decades. As an active IT industry thought leader, I am particularly interested in enterprise digital transformation, globalization, and generative AI. How We're Disrupting/Improving the Cloud Data & Analytics Industry Nearly plan to invest in GenAI in the next 24 months, and we are launching solutions – BlueCasual and BlueInsights – that specifically help non-technical users access data and streamline processes through an LLM-based chat interface. 63% of marketing leaders Currently, the vast majority of marketers aren’t using data and analytics to determine campaign ROI because advanced insights are still interpreted by engineering and analyst teams. The co-innovated solutions of BlueCasual and BlueInsights include causal inference machine learning (ML) and a Generative AI (LLM) chat feature, providing digestible data that allow IT teams to deliver greater returns to their stakeholders at scale. This cutting-edge accelerator tool utilizes the Snowflake Data Cloud and an advanced machine-learning model jointly developed by its teams. It provides a more comprehensive perspective on ad performance and reveals statistical insights about campaigns by leveraging first-party, third-party, operational, and sales lift data. BlueCausal enables precise measurement of impact on sales, incorporates confounding variables, and analyzes the response of effect variables to optimize campaigns effectively. Combining internal operational and revenue data with external data in Snowflake offers valuable insights to advertisers, marking a significant advancement in ad campaigns. BlueInsights complements BlueCausal to help non-technical users uncover data insights, comprehend complex statistical information, and take action through a simple chat interface. This Generative AI solution ensures that those unfamiliar with data science or advertising jargon can easily leverage advanced machine learning models to improve business outcomes. Standing Out From The Crowd In the age of remote work, we have embraced a "BuildGlobal" Borderless Delivery model. This innovative approach provides access to top technology talent worldwide, allowing us to implement practical business strategies and technology solutions for clients and our team to work from anywhere, regardless of geographical constraints. A borderless model enables us to streamline day-to-day operations while focusing on engaging employees, fostering a positive company culture, and promoting a shared long-term vision for success. We're also expanding our artificial intelligence and Generative AI capabilities to bring advanced predictive solutions and real-time analytics to clients across all business functions. Our enterprise GPT solutions prioritize optimization, observability, and provide detailed predictive capabilities. With Generative AI, users can now ask questions that previously required a data scientist's expertise to enhance their business outcomes. Recently, we’ve been working toward making advanced machine learning accessible to businesses through first-class partnerships with Snowflake and AWS. Another differentiator includes our senior talent acquisition and retention from respected companies like AWS, Oracle, Slalom, and ThoughtSpot during widespread tech layoffs. Our Predictions/Thoughts on the Cloud Data & Analytics Industry in 2023 Data and Analytics are essential in navigating modern business challenges across various sectors. Whether a company aims to cut costs, streamline processes, or expand, leveraging data and analytics is crucial in driving these improvements. Data analytics will increasingly focus on data freshness for real-time analysis, providing companies with prediction capability, better decision-making, and a competitive advantage. Additionally, AI and Machine Learning technologies will allow companies to analyze unstructured data faster and more effectively. Many businesses are now adopting a multi-cloud or hybrid cloud strategy, incorporating multiple cloud providers or blending on-premises infrastructure with cloud resources. While deploying multi-cloud environments can be complex, this approach allows companies to tailor solutions to their specific needs and access a more comprehensive view of their data. With this, companies will start working with multiple cloud providers specializing in specific solutions to optimize cost, speed, and performance. As more companies move their workloads to the cloud, we will also see an increase in cloud-native technologies and practices, including containerization, micro-services, and infrastructure as code. Additionally, cloud providers' advanced AI and machine learning services further enhance automation, predictive analytics, and machine learning opportunities, enabling enterprises to focus on optimizing customer experiences. Cloud computing and data analytics will continue shaping the enterprise software services market for the foreseeable future, providing scalability, flexibility, cost-effectiveness, competitive advantage, real-time insights, and improved collaboration. With this, deploying Artificial Intelligence in digital transformation initiatives will help monitor and manage back-end cloud resources, freeing IT workers to focus on human intelligence, such as the customer experience. At the same time, AI's ability to process data to inform better executive decisions and processes will become a top investment for enterprises looking to increase productivity and revenue over time. What word defines the state of Cloud Data & Analytics in 2023? Generative AI is a big topic right now, and for a good reason. GPT solutions for enterprise teams are coming. IT companies are creating versions of ChatGPT trained on internal systems for better business intelligence and performance. This will be a game-changer for enterprises that can deploy internal Generative AI solutions because it democratizes insights for teams to stay competitive through customized platforms. Why we decided to participate in HackerNoon's Startup of the Year awards We are entering a new growth phase with the recent explosion of AI and increased demand for data and analytics implementation services. BlueCloud is uniquely positioned to create enterprise Generative AI solutions in collaboration with data partners like Snowflake. A huge priority is diversifying our customer base by adding more mid-market and Global 2000 customers to our roster and creating significant customer projects with them. Another target is expanding existing client accounts and our investment in those growth areas that have shown early promise, including enhanced investment in Industry-based and technical solutions with Snowflake. Our strategic partnerships have positioned us well in the market, and we are investing heavily in those relationships to diversify our client roster for steady growth. BlueCloud is on a path to becoming a hybrid product and service organization. We’re trying to create additional solutions that deliver value to customers quickly and continuously maintain high-quality service, bridging the gap between technology and delivery and value. Final Thoughts The success of OpenAI’s ChatGPT and other large language models was only made possible by the advancements in cloud computing and the availability of data, which means that IT services companies like ours are at an advantage to create Generative AI solutions that enterprises need to stay competitive. We are enabling cloud data platforms like Snowflake as enterprise data sources for AI and making advanced ML more accessible to business users. Many organizations are leveraging GPT for SQL query generation; however, we take this a step further to support true personalization and business outcomes. Vote for us today!

Startups of the Year 2023: BlueCloud - What The Global Digital Transformation Company Has to Offer

Data self-service, the ability for stakeholders in the organization to answer their own business questions with data, is a top initiative for nearly every data leader I’ve spoken to this year. It’s so foundational to creating a data-driven organization, that most of the questions surrounding its focus on the “when” rather than the “why.” That’s why we were surprised to hear it became such a passionate debate at our New York IMPACT event, which included some of the area’s top data leaders working on some of the most complex challenges facing our industry today. When we asked a member of our panel what they were struggling with the most, a data engineering leader at a multi-billion dollar company with a modern stack said: “I am drowning in from the business.” ad-hoc questions They weren’t alone! Everyone in the room started scribbling notes–clearly the panel had struck a nerve. So we posed the question, “is self-service a lie?” It immediately divided the room and evoked strong opinions. Even on our three-person panel, we were deadlocked with one person agreeing, one disagreeing, and one rejecting the premise of the question entirely. This is an important question with real ramifications as self-service is at the core of two of the hottest trends in data: the and the decentralization of the data team. data mesh In this post, we will showcase both sides, and leave it to you, dear reader, to determine what makes the most sense for your organization. Let’s start by hearing from the prosecution. Self-Serve Is A Lie The argument goes something like this. The number of questions that people can come up with is truly infinite. The very nature of data, and really the scientific method in general if you want to get grandiose (we do), is the minute you present the answer to one question, it sparks another. And for those who have spent any time working on a data team, you know the inevitable way the business operates is that an executive needs an answer to a very particular question –and you are the path of least resistance. now Of course, the inevitable emergence of the data question fire drill is not entirely the fault of our data-consuming colleagues. Let’s be honest with ourselves. Data engineers have a love / hate relationship with being a . data wrangler For example, one customer told me, “10% of my job is answering questions about our data and I hate that part of my job” while another one said they really enjoyed being the go-to source for data decisions. It’s a matter of perspective and degrees. No one enjoys being interrupted and being asked a million questions or . really treated like a data catalog On the other hand, deep down, do we love being the smartest person in the room? Do we take some professional pride in being the go-to star who can swoop in to put out fires and save the day? Does that stunt some of the natural inclinations of data consumers toward self-service? sometimes Well, it’s complicated. But, perhaps the most compelling argument regarding self-service as one of data’s biggest lies is the potential for it blowing up in your face. Everyone who has worked in data has had that moment where they thought they gave a consumer enough context to empower them to complete their project, only to then have that person do something completely unexpected and egregious with the data (with the best of intentions, of course). So instead of spending 20 minutes directly holding their hand, you pointed them in the self-serve direction and now you’re working late re-doing their work. For example, my colleague worked at a company with a CEO that was very involved with the data, which was a tremendous benefit. As they grew from a 50 to a 500-person company, the CEO by necessity was pulled in many different directions and couldn’t be as close to the data as they used to be. When the CEO (admirably!) went to do a bunch of analyses directly and pull data for board slides, they hadn’t realized the data they once knew well had changed. The slides ended up pretty far from reality and had to be redone just hours before the board meeting. This type of story is common and can happen with even the most benign and standardized data. For example, could have a batch of data that accidentally double surveyed the same people or had the wrong time window due to one large historical load into a table creating wrong timestamps. Or they could simply be using an older, deprecated table. NPS survey data Self-Serve Is The Truth Now it’s time to hear from the defense. The argument goes something like… If perfect is the bar you need to clear, then you will never start any data initiative. And just like your will never be perfect, neither will your self-service mechanisms. And despite this realization, you still need to make the effort, otherwise you’ll have larger problems on your hands. marketing attribution formula If you don’t at least try to implement a self-service motion, you will need to start initiatives from scratch every time you lose a data engineer with that particular bit of tribal knowledge. That can’t scale. Besides, it’s not just the destination that has value, but the journey itself. Along the self-service path are the pitstops of data documentation, literacy, discovery, and quality. All incredibly important to an organization in their own right. We’ve also seen that centralized data teams are far from perfect. Often it’s the domain analysts who are closest to the data who can be the most effective and wring the most value from it. Finally, if there was ever a time to invest in self-service it would be today with all the tools available across the modern data stack. The cloud data warehouses and lakes make data access easier than ever while tools, like platforms, that automatically collect and surface up-to-date lineage and metadata provide more context than ever before. Tools such as and truly democratize data so the less technically skilled stakeholders can model and query data without the use of code respectively. data observability dbt ThoughtSpot Not only is self-service possible, but it’s an absolutely necessary exercise for the organization to agree to common standards, definitions, and metrics so they can actually become data-driven. Verdict: Iterate and let the business help you help them With self-service, as with many things in data, it’s about the definitions you create and where you draw the line. In fact, it’s rarely the case that self-service needs to be defined as every person in the company being able to answer every question they can think up without having to involve anyone in data engineering. If that’s the case, where do you draw the line between those that need a higher level of self-service or direct data access from those that don’t? The consensus that emerged on the panel was data analysts are going to form the bedrock of your self-service and data democratization efforts. It’s important they are or They need to be both equipped and empowered to answer the questions they field as they sit closest to the business. become fluent in writing SQL. Unfortunately, there is often a wide gap between the engineering and analyst teams. Ironically, self-service initiatives can actually exacerbate that distance and so careful forethought needs to be given to the process of ensuring data producers and engineers are still communicating and staying close to their customers. One popular strategy for bridging this chasm is to leverage analytics engineers to serve as the layer between your messy, behind-the-scenes production database and the data consumer-facing data warehouse or instance where the cleanest, most documented data lives for wider exploration. Another solution is to train data consumers to leverage what is typically an unused goldmine of information about a company’s data: Slack. A quick Slack search for a table name before asking a question can save tons of time. We also recommend your self-service platform and customer base start small. On the platform side that could mean certifying tables for self-service or leveraging a . data product framework On the customer base side, just like self-service doesn’t require everyone at the company to write SQL queries, it also doesn’t mean everyone across the data team needs the same level of service. As former UCLA basketball coach John Wooden said, “Fairness is giving all people the treatment they earn and deserve. It doesn’t mean treating everyone alike.” In other words, there are going to be some domains that are more mature and require more self-service capabilities than others. Start with one team, make them deliriously happy, and scale from there. Finally, another self-service approach worth considering is to flip the traditional paradigm on its head and start with your data consumers first. Work with them to understand their needs, write a , create data SLAs and deliver the data pre-modeled. data contract Ultimately, if we MUST preside as the judge in this courtroom, we’ll declare that self-service is a worthwhile endeavor and an obtainable goal for most data teams. But, we’ll also concede it’s a good idea to understand that reality will dictate those last-minute ad-hoc requests will never be eliminated and it’s better to have a process to accommodate them than force everything into a self-serve world. What do you think? This article was co-written with Francisco Alberini. Also Published Here

Can Your Organization's Data Ever Really Be Self-Service?

Reporting tools organize scattered data and present them in the form of tables, charts, and dashboards. Their role is essential because it transforms abstract data into intuitive information, shows past and current status, and makes finding trends and patterns among much quicker. In this way, it is easier for companies and departments to track business progress and adjust strategies timely. big data Before the advent of professional reporting tools, Excel was the main tool used to build reports. But people who often deal with reports know that it is very troublesome to use Excel as a reporting software. It is easy to make errors and is time-consuming. Modern reporting tools more and more to meet the actual reporting needs of enterprises, from data collection, processing; to report making, presentation and sharing. Meanwhile, many reporting tools are designed for different business units, such as finance, sales, marketing, etc. In this article, I’ve reviewed some of the most popular reporting tools on the market today and picked out the ones that perform best in specific features. Welcome to take advantage of it! What Do the Best Reporting Systems Have in Common? The best reporting tools on the market have a few things in common. Before we start, here’s a list of the top 4 critical functional aspects of reporting software to guide you in your decision. 1. Convenient data combination The best tools usually support broad data sources. But that's not enough for enterprises. The reporting value also relies on getting scattered data currently housed in various systems into one report to break data silos and give users a comprehensive view of business performance. Best reporting tools on the market offer an easy way to combine data via a few clicks, handling multiple sets of data in a single report. 2. Ease of use Whether the IT department or the business users, they all want a reporting tool to help them get rid of the tedious work of making reports manually with Excel. In this way, they can spend more time analyzing and mining the meaning behind the data. The best reporting software offers an intuitive visual design environment for users to create custom reports via drag&drop. The learning cost is low. 3. Flexible to integrate and customize A good reporting tool should be flexible enough for developers to integrate the reporting platform with other systems. So users can check the reports directly in the CRM or ERP system. It removes the hassle and improves the user experience significantly. It also offers web scripts or API interfaces for deep development and rich reporting systems features. 4. Multiple reports delivery ways Sharing reports is often the last but most crucial step in an organization. It presents to your team, leaders, or clients the past performance results and the data analysis you have done to drive the next step or strategic change. A good reporting tool provides various ways to export and print reports, also supports the automated push of reports by email, SMS, or communication software. List of Best Reporting Tools and Software Based on the best reporting tools criteria, I selected eight top reporting softwares, ranging from free and open-source to commercial, specialized in different areas. - Best free reporting tool Google Analytics - Best reporting tool for enterprise-scale solutions FineReport - Best open-source reporting tool Jasper Reports - Best reporting software for data visualization Tableau - Best reporting tool for analytics ThoughtSpot - Best reporting software for financial Sage Intacct - Best reporting platform for marketing AgencyAnalytics - Best tool for project performance reporting Celoxis An In-depth Review of These Dashboard Reporting Tools 1. Google Analytics - Best free reporting tool If you work in marketing, is no stranger to you. Google Analytics It allows marketers to monitor users’ behavior on their website, social media, and other campaigns through real-time online reports. Google Analytics offers a free but robust entry-level version where you can track your digital marketing performance such as traffics, acquisitions, referrals, bounce rate. You can also such as lead form submissions, email list sign-ups, purchase completions on the report. The data is real-time so that you can adjust your campaign strategy timely. set customization goals You don’t need to worry about the deployment and operation if you have no background because the It is also convenient to export the reports into Excel and PDF. Or set scheduled reports to share with your team via email. reporting platform is intuitive and user-friendly enough. Finally, the feature provides you more comprehensive data analysis of your website performance. It would alert you if your web page took significantly longer to load. Analytics Intelligence the free version is powerful, supports customization goals, intuitive interface, analytics intelligence version Top features: the basic version is free Price: 2. FineReport - Best reporting tool for enterprise-scale solutions is an enterprise-scale reporting tool that provides an advanced solution for various, frequent, and complex business reporting needs. FineReport The design of FineReport covers different roles’ needs in the reporting. you can’t miss FineReport’s For IT departments, simple deployment and integration. FineReport is written in Java, which of enterprise software. It is flexible to integrate the system with third parties such as CRM, OA, ERP, etc., and embed reports seamlessly into any application software. ensures the flexibility and scalability FineReport’s also simplifies the customization process of reporting systems. Besides, . rich APIs the operation and maintenance cost is low FineReport’s provides the best platform for data extraction, integration, visualization, query, and analysis, without the need for code lines. For report developers, interactive Excel-like interface The data entry feature is outstanding. Users , with functions such as data validation and temporary storage. This streamlines the data collection process in reporting. can write the real-time data back to the databases via the online reports it is easy to get business insights from cool reports and dashboards on the . For managers, all-in-one management platform On top of that, managers can in the meeting. view reports no matter on the mobile devices during the trips or on TV screens simple deployment, flexible enough to integrate, customize and extend, data entry, scheduled report, TV and large screen display, 3D charts with cool animations, multiple formats exportation. Top features: free for personal use, quote-based for enterprise use Price: 3. Jasper Reports - Best open-source reporting tool is a Java-based engine that is the Jasper suite’s core. JasperReports The open-source community edition includes broad components for reporting. provides Java classes and APIs that allow users to generate reports and visualizations based on designs. It is available standalone and also comes embedded inside JasperReports Server or JasperReports Studio. The JasperReports Library is a standalone and embeddable reporting server that embeds the JasperReports engine, enabling users to operate on the user-friendly, web-based interface. The JasperReports Server Besides, JasperReports for users to explore the data. offer data integration, ETL, and OLAP components All the open-source reporting tools offer essential elements such as lines, areas, pies, radar, ring, etc. However, Jasper Reports collection among them. It adds maps, sparklines, pyramids, and Gantt diagrams provides the richest visual elements in the library. Besides, Jasper Reports , with a stable medium activity by its editor TIBCO Software. has a well-maintained repository offer data integration, ETL, and OLAP components; provides the richest visual elements among all the open-source reporting tools Top features: starting from $233.00/year, charged by per feature Price: 4. Tableau - Best reporting software for data visualization is considered one of the reporting tools with the most appealing visualization effects that make your data visible in a unique, and compelling presentation. Its compared to competitors. The visualization dashboards in the Tableau gallery are always impressive. Tableau visualizing capabilities are unparalleled Besides, its is on the top list of Tableau’s benefits. It provides a for users to easily understand data and discover in-depth insights. And the operation is fast enough, even process the big data stored in various places. robust and reliable performance highly interactive interface When you launch Tableau, the data connectors are readily available, which allows you to connect to any database. Then, the pulled raw data can be arranged into catchy diagrams via drag and drop in Tableau Desktop. The reports and dashboards you created can be shared in the Tableau server, where licensed users can access and use them. Apart from sharing, this enterprise platform also supports collaboration, distribution, governance, security model, and automation. Tableau Server also . You can ask a question with natural language and get answers in the form of a viz. And AI explained data points based on advanced statistical models help you make AI-driven decisions. leverages the power of natural language and AI Besides, Tableau where you can beef up your knowledge in data reporting, visualization, and business intelligence. has a thriving community unparalleled visualizing capabilities, robust and reliable performance, highly interactive interface, natural language and AI tech, online community Top features: Tableau’s pricing is available in three subscription options. Price: 5. ThoughtSpot - Best reporting tool for analytics is a BI reporting platform with solid search and AI-driven analytics capabilities. It brings a for business users to quickly ask questions through , regardless of the dataset’s size. ThoughtSpot Google-like search bar natural language queries It leverages the , which can continually scan billions of data to find the answers, even automatically AI insights engine SpotIQ discover the insights that you might not have thought to ask. ThoughtSpot also provides a user-friendly platform for users to create reports, build dashboards and analyze data without relying on IT. It is easy for non-technical business users to interact with ThoughtSpot and gain insights through anomaly detection, trend analysis, comparative analysis, and more. Besides the mainstream databases, ThoughSpot Cloud also supports popular like Amazon Redshift and Snowflake, allowing you to connect with and combines data sources on-premise or in the cloud. cloud data warehouses Google-like search bar, natural language queries, AI-driven, analyze billions of rows at sub-second speed, cloud data warehouses support Top features: available upon request Price: 6. Sage Intacct - Best reporting software for financial With business growing, companies have unique and sometimes highly complex accounting and financial management challenges. Accordingly, the demand for flexible, in-depth financial reporting is increasing. is a that can transform the way your financial reporting and business works. Sage Intacct cloud-based financial reporting tool It provides an easy-to-use solution for building customizable reports that incorporate financial and non-financial data, and the ability to quickly slice and dice the details. offering real-time filters Sage Intacct also supports such as accrual, cash reporting, tax basis, etc. Besides, via Sage Intacct, you can such as accounts payable, month-end close, and expense management. multiple accounting methods automate the business process Sage Intacct’s technology , so it’s easy to connect your financial solution with other cloud applications. leverages open APIs But you don’t need to worry about your data security. Sage Intacct offers , giving you a secure environment to protect your financial data at a low cost. world-class disaster recovery and security backups offering real-time filters, multiple accounting methods, automated accounts payable, month-end close, world-class disaster recovery, and security backups Top features: available upon request Price: 7. AgencyAnalytics - Best reporting platform for marketing is a marketing campaign-focused reporting platform where you can put all of your marketing channels in one place to get the whole picture of your campaign process. Agency Analytics It is highly suitable for marketing agencies to track marketing campaigns and report results to their clients. Agency Analytics supports including the most commonly used marketing tool such as Google Analytics, Search Console, Adwords, Facebook Ads, social media. 50+ data source connectors, Besides, AgencAnalytcis such as track the keywords ranking, monitor the backlinks, and audit the website. That means, if you don’t have high requirements for SEO monitoring, extra SEO tools are not necessary. offers the basic features of SEO tools After the reports are generated, you can choose to export reports into beautiful PDF or send automated emails to your clients. Last but not least, Agency Analytics . It is user-friendly for your global customer to operate the system in their local language. supports 23 languages 50+ data source connectors, offers the basic features of SEO tools, automated reports, broad language support. Top features: $49-$399/monthly, 14 days free trial Price: 8. Celoxis - Best tool for project performance reporting Using reporting in the project can help you stay on top of your projects and manage your stakeholder’s expectations. You can also keep your team members on the same page, let them know your project plan and status. is a project management software but stands out for its reporting with its advanced reporting engine. Celoxis Besides the such as project planning, project tracking, resource management, team collaboration, Celoxis provides features that improve your project’s bottom line. basic features of project management software real-time financial management In terms of reporting, Celoxis has the , where you can implement cross-project reporting, drill-down charts, and automated reports. best analytics engine in the market today It offers both in the cloud and on-premise versions. simple and straightforward interface, SaaS plan, comprehensive project management features, real-time financial management, advanced reporting engine Top features: $25/user a month for the SaaS plan, or $450/user for one-time payment on-premise. Price: Other Reporting Apps Options - a traditional reporting system for SQL, support 28 languages 1. Crystal Reports - a web-based reporting generator, provide automatic access to SAP BW queries 2. Cognos Analytics - an online reporting tool from Microsoft that works well with Excel 3. PowerBI - a hybrid-cloud bi reporting platform, the in-chip architecture accelerates the time it takes in data ingestion. 4. Sisense - an innovative BI reporting platform that combines visualization, machine learning, and collaboration. 5. Yellowfin - a self-service reporting tool with an “All-in-one” design that includes performance management 6. Board - a SaaS platform for building reports and dashboards 7. Cumul.io - an embedded analytics reporting system, support ad-hoc reporting 8. Izenda - provides finance-owned solutions that include a reporting tool 9. InsightSoftware - an open-source reporting and business intelligence tool, offer a Java environment 10. Birt

8 Tools To Improve Business Reporting

Geo is practically everywhere today! With the prevalence of GPS all around us & the rise of smartphones, location data is being collected in abundance. Legend almost all of the data (~80%) that companies store has a location component. says Our lives are now filled with products and services available at the tap of a phone, getting things delivered in minutes to our doorstep! Today, it is hard to find an app which doesn’t ask for your location permissions. There is a famous in the GIS (Geographic Information Systems) world which says When you place an order from any food delivery app, every event starting from placing of the order — to delivery executives (DE) getting assigned to — the DE picking it up from the restaurant — to him delivering happens at a certain lat-long! quote “everything happens somewhere.” Which brings me to my next point: Why are we obsessed with “location” as a dimension? Firstly, there is no “mobility” without dynamic location. In other words, location is a fundamental aspect whenever there is movement of assets (people, vehicles, cargo, parcel) on the ground. After all, in cities and towns where things change at every sq km in a city, it’s important to add the “context” of those areas. For the on-demand economy that we live in, analyzing location data across all your dimensions (users, stores, partners) becomes in real-time. This is because you have to match your supply with demand, do location-based pricing, promotions or faster and more accurate deliveries at scale! critical And finally, the most successful for consumers are based on sequential activities at a very granular level: Where are you? Where do you need to go? What do you want to do when you get there? This becomes important when you need to acquire customers by targeting ads and retaining and engaging them by their behavior. location-based experiences studying Location is not only about a point on a map. It is about a line. It is about movement. But all this time, the location component has been particularly in business decisions. This is because conventional analytics tools are fantastic for the treasure trove of statistical data that businesses have but somewhere, they fall short of . neglected location-based decisioning You might be wondering: “But, why?” Before we deep-dive into that, let’s understand how geospatial data is different from statistical data. Location is exciting! If played right, it can drive significant revenue improvements (case in point: Uber & Airbnb). Its high usage and real-time nature make it really valuable and sticky. What sets geospatial data apart from statistical data? Just like text, sound, image are different kinds of data, Latitude-Longitude is a different kind of data that can add immense depth, meaning and insights to statistical data in a space-time context. Statistical data comes in a tabular format and usually, is comprised of two elements: values across time. (also called geographic data or location data) is often formatted as points (latitude-longitude coordinates), polygons or polylines. Geospatial data It incorporates the third dimension on top: values across time and dynamic location — which requires a completely different approach and treatment. Computing metrics for static assets involves plotting points on a map and calculating metrics for that. Movement analytics concerns itself with how do we visualize, analyze and optimize how things move on the ground! Two special properties of geospatial data are autocorrelation and non-stationarity, which make it difficult to meet the assumptions and requirements of traditional (nonspatial) statistical methods, like OLS regression. means how similar (correlated) are my measures between nearby observations. It falls in-line with the first law of geography as well- Spatial Autocorrelation “ everything is related to everything else, but near things are more related than distant things ”. is a condition in which a simple “global” model cannot explain spatial relationships between the variables. The nature of the model alters over space and behaves differently in different parts of the study area. Spatial nonstationarity Let’s answer our pending question now: Why are current statistical tools not apt for spatial analysis? Today, the current way of dealing with location data inside companies is broken. Due to the dearth of any location analytics products out there, most of the businesses have no choice but to rely on traditional BI and analytics tools. It’s not puzzling that this is a highly ineffective strategy because these tools are not really meant for geo-analysis. 1. Formats Most companies collect location log data in the form of . The way to analyze the patterns they encompass is to have an infrastructure that can ingest these pings in real-time — something that platforms like Periscope or Tableau don’t cater to. “ pings ” Moreover, location data is present across disparate databases, in different structures. Hence, slicing and dicing variables across tables in real-time comes even more complex. 2. Visualizations All of us know that a statistical dashboard contains bars and charts which sprout from carrying out operations (sum, count, divide, average, etc) on variables. While getting live trend updates through spikes and dips on graphs might be helpful, these charts are work better on aggregated historical data. Adding or dividing lat-longs and creating bars and charts on them is pretty futile. To make sense of these lat-longs, you need to have a map by your side to understand their spatial distribution! Another important aspect that governs all the properties of a geospatial dashboard is a . Not having this layering mechanism which you can use to display multiple types of data points on the map, sort of misses the point. For instance, layer layers help in viewing how my orders (first layer) and partners locations (second layer) are distributed across my area clusters (third layer). Maps are also more insightful to draw inferences than bars and charts if there is movement of components on the ground involved. Hence, real-time geographic analysis when everything is dynamic becomes fundamental. 3. Aggregation Traditional BI tools like and offer the capability to plot points on a map. However, just plotting points on a map is not adequate as billion dots in space is not very intuitive. Moreover, PowerBI Geckoboard Location intelligence is so much more than tracking and plotting points on a map! Strategies like clustering, heat mapping, aggregation, indexing, etc. come in handy to absorb a large number of points. Some tools like and also allow the creation of heat maps — a fantastic way to depict the patterns of metrics. The disadvantage of heatmaps is that they are only a visual representation, thus restricting you to do any sort of analysis on it. Tableau Periscope To know more about why heatmaps don’t work, check out! this A more efficient way to do aggregation is indexing them on or . Once you analyze the pattern of your metrics (such as demand and supply) across grids, you can use the cells as a single unit in your models as well. Indexing also helps to go very granular in your analysis. hexagonal grids geohashes 4. Data Preparation Using and , you can clean your statistical data by taking care of the blanks, spaces, data formats, NAs etc whereas cleaning of GPS data involves snapping it back to the nearest road or correcting for . (You must have observed often while using Google Maps, GPS goes off very far randomly.) It is safe to say that GPS as a technology still has miles to go! Cleaning: Periscope Thoughtspot spatial outliers Platforms like or allow you to merge two tables using an inner join, left join or a right join on the basis of a common identifier. It’s quite difficult to do spatial merges using these platform if, for instance, you have data across three dimensions: users, delivery partners, and stores (sometimes they come in different formats). Merging: Tableau Prep Metabase A involves inserting columns from one feature table of a layer to another in the spatial perspective. spatial join For example. merging a Kormangala area polygon with all the ride pick up points inside it. Enriching of any data implies adding new layers of information and merging it with third-party or external data sources. In the GIS world, we enrich spatial data for a better context of areas in which the points are present. This means adding the environmental layer of roads, traffic, weather, points of interest, demographics, buildings, etc. Enriching: The internal solutions not working out? Some companies, of course, realize this and hack around open-source tools (like or ). But these open source tools come with their own list of constraints and limitations. Kepler.gl QGIS However, the issue doesn’t get resolved here because geospatial data itself comes with a bucket full of challenges. Performing geospatial queries on streaming data become very compute-intensive and legacy technologies (like ArcGIS) provide very little support. The complexity increases with visualizing large geospatial datasets with any sort of interactivity at scale. Sometimes developers also build their own internal tools, but most of the times they are not well suited for all different audiences inside the company. Since the tools are not built in a scalable way, maintaining these suck up a lot of developer bandwidth often! A lot of times there is even a repetition of effort and the wheel keeps getting re-invented over and over again. As Jeff Lawson from Twilio — “It is easier than ever to build software but harder than ever to operate it”. said Why are we building Locale.ai? It all started with a personal problem. As data scientists working with geospatial data, the existing analytics products were futile in our daily workflows. Hence, we had to build our own tools and libraries for our everyday workflows. We then realized data scientists around the globe face similar problems when it comes to location data. As a result, businesses are struggling to attain operational efficiency! At , we plan on solving these problems once and for all. We want to make all of these processes less painful and building a world where it is very easy to get all your geospatial answers in minutes! That’s why we are going back to the drawing board and handcrafting this experience completely from scratch. Locale So, the next time you want to order medicines in case of an emergency, you won’t hopefully read on the screen, “ ” Next time the delivery guys won’t have to stand idle in the scorching heat, cold or rains waiting for the orders to come. Delivery guys are not available. Please check again later. They can be incentivized to move to high demand areas and can earn more money. The push notifications that you get won’t be spam — they will be shot to you at the right place and right time. Read Similar: Bridging Supply-Demand Gaps in Last-Mile Delivery Companies Geospatially [ ] Link Carto vs Kepler vs Locale: Which product to use for geospatial analytics? [ ] Link

Why Do We Need a New Product for Geospatial Analytics for Mobility

Happy New Year!  2019 has come and gone like Kylo Ren’s reign in The Rise of Skywalker, and so it's time for my annual prediction piece. But first, a quick look at the score card for my : 2019 predictions 2019 saw numerous trials, including at carnivals in Nice, France, airports in Las Vegas, and public housing in Detroit. Of course bans of the technology has already occurred, too, in cities like San Francisco and Somerville, MA. Facial recognition trials will launch in large public venues outside of China. Grade:  A I predicted this space would reach a third of the value of the workflow platform space by the end of 2019. With Databricks hitting $6.2 billion in value, ThoughtSpot achieving a $2 billion valuation, and many others, I nailed this one. Enterprise analytics will become red hot. Grade:  A I said we'd see "major automobile manufacturers actively acquire or sign new partnerships with various tech companies to avoid becoming irrelevant in the near future." Hyundai's $4 billion self-driving joint venture with Aptiv, Toyota’s $100 million fund for autonomous driving, and Daimler and BMW's partnership to develop autonomous driving technologies are a sampling of the moves made in 2019. Car companies will begin transforming into tech companies. Grade:  A+ Hear those ads?  (chirp, chirp). Voice advertising will be stalled. Grade:  A+ 2019 saw headlines like “Data science dominates LinkedIn's emerging jobs ranking," but I believe this trend needs to ferment a bit more like a good batch of kimchi. It takes time to train and develop new waves of data scientists. According to LinkedIn, over the past three years data scientist jobs experienced a 37% hiring growth. I also know of a large pharmaceutical company that laid off over 1,000 research scientists and hired nearly 500 data scientists, but these type of hiring trends aren’t strong enough to call them the new “call center reps." Data scientists will become the new call center reps. Grade:  B- I also want to revise the grade on one of my 2018 predictions. I gave myself a F after my bet that China and five other OECD nations would fell flat. on that one. I'm confident many other nations will quickly follow China’s move. National cryptocurrencies. launch their own cryptocurrencies in 2018 But China did come through in 2019, so I’m going to bump my score up to a “B” This coming year will be filled with more exciting advances and changes in the tech industry even with an economic downturn looming, so here are my predictions for 2020: What we can look forward to in 2020 1. Cloud kitchens will become a multibillion dollar industry Launching a new restaurant in the largest cities is riskier than launching a tech startup. The restaurant business is a in the U.S. alone and is begging for efficiencies. " " -- virtual restaurants that cater solely to delivery -- have emerged as a solid trend. They eat up far less real estate costs than traditional restaurants and capitalize on the many new on-demand food-delivery companies, such as UberEats, Deliveroo, and . I see taking place. (A cloud-kitchen startup that was in my accelerator program in Korea was recently snapped up in an acquisition just 6 months after our investment, despite me trying to convince the founder to ignore the offer.) $799 billion industry Cloud kitchens their ilk big opportunities within this space in 2020, with a couple of unicorns being formed and a few acquisitions 2. Gibson’s and Chicago Cut will serve Impossible Food’s meat substitute To be honest, I’m not a fan yet of Impossible Foods or Beyond Meat, but my taste buds do not matter with this prediction. The real prediction is that these plant-based meat substitutes are becoming so pervasive that they will infiltrate two of my favorite steakhouses in the world. in the U.S. The drivers are climate change and the practical fact that these newer meat substitutes taste good to meat lovers. For 2020, I predict meat substitutes will penetrate over 10% of restaurants 3. 5G will sputter searching for the right services Beyond the hype of 5G, there are limitations to the millimeter wave-based network part of 5G and many network build outs are behind in the U.S. Regardless, the most important aspect about 5G's adoption is content and services. Games on phones and VR headsets (i.e. Oculus Quest, Hololens) with 5G chips are a given, but what else? There will be adoption, but not a mad rush in 2020 for 5G devices and services. The real rush won't come until more services that fully utilize 5G appear. 4. We'll see a global movement towards a cashless world Even with all the talk about Alipay and WeChat Pay, , but globally we are far from a cashless world. Even last year, when speaking at a conference in Berlin, I had to walk five blocks to find a store that accepted credit cards to buy a bottle of water. China uses cash more than the U.S. (40% vs. 37%). South Korea is the lowest at 14% But we will get there because the younger generation is driving this cashless world (i.e. Venmo), and the technology is finally good enough that people can easily forgo cash. This year will see various online payment systems (PayPal, Crypto, Apple Pay, Google Pay, etc.) pick up bigger adoption, shifting the average cash usage of the 10 countries listed here to below 45% (vs. 55%): And beyond .... Since we're also at the start of a new decade, here are a few more predictions of what the world will look like by the time we hit the next decade in 2030: 1. Technology will bring back suburban development Discussions of trends for denser urban environments are almost becoming like what diet has the most impact… Keto saved my life!  Plant-based is the way to go! Mediterranean diet is the game changer! (I just started the Potato diet, so do not send me pictures of ribs, bacon or filet mignon.) So my prediction here is one more addition to the current mix of ideas of where urban development is trending. I believe that the development of autonomous vehicles (e.g. cars, buses, trucks), cloud kitchens, , and other new technologies will shift the population trend and real estate development back towards suburbs and even to the outskirts of major cities over the next decade. The concept of commute time will become different with autonomous cars since people can sleep more, work, do home finances or whatever they please while in transit. A welcome outcome of this reverse mega-city trend would be the stabilization of or decrease in real estate prices, especially in cities such as San Francisco and New York. drone delivery 2. Technology tribes will begin to coexist with nations scares the bejesus out of nations and threatens their sovereignty, so during 2020 they continue to fight like a cornered rat against its launch. But the very concept of Libra signals the beginning of something new: the development of technology tribes that exist outside of sovereign nations -- or coexist with them. Facebook’s Libra As technology becomes more ubiquitous and the digital divide closes, new generations will develop stronger affinities with specific tech communities than with ethnicity or race. The coming decade will show beginnings of this trend, but it will become really palpable 20 or 30 years out. 3. The next Mark Zuckerberg will be from outside Silicon Valley Until recently the tech world has been divided into two groups: Silicon Valley and everyone else. But the world is becoming flatter now in terms of entrepreneurial knowledge and execution. The venture capital and investor quality along with Silicon Valley’s “pay it forward” culture will strengthen across other global startup ecosystems over the coming decade to the point where we will see the next Zuckerberg rise out of what we currently deem a second-tier startup city. is cofounder and Partner at , a network of accelerators and venture capital funds. Bernard Moon SparkLabs Group ( originally published in VentureBeat )

2020: Our Meatless, Cashless, City-less Future 

Need to find the best Artificial Intelligence/Machine Learning companies in India? Whether you have noticed or not, we all are living in an era driven by Artificial Intelligence. Firms are footing the bill for billions of dollars just to have a competitive edge and stay ahead of the ever-changing dynamics of the digital landscape. And, if you are not AI technology compatible, the accord predicts you will soon fade from the industry. After all, you will need to hire the best AI developers to stay relevant in today’s instant changing technology scenarios. And, this got me thinking - to bring out the top-most Artificial Intelligence/Machine Learning development companies in India that are re-inventing the global revolution? Undoubtedly, it is an interesting quest. To assist you with the right AI development approach and company, I put down a framework or scrutinized criteria to funnel the topmost Artificial Intelligence company in India. (Scroll to know more) Hopefully, this blog will prove a boon in driving you to the right destination. Similarly, highlighting the top AI investor in your region, that are harnessing the AI/ML technologies for a better future for everyone among us. So, are you ready to dive in and have a look at the criteria I have created to short-list the leading Artificial Intelligence companies in India. The below list has been curated and evaluated on imperative parameters that act as indicators for the successful launch of your project. - Google rating by clients and organic ranking overview - Full money back guarantee - Customer reviews from Goodfirms, Clutch, and Glassdoor - Diversified Portfolio - Technical expertise on latest software trends - Client retention rate of the company - Total number of successful software projects - Personally talking to their clients Here is the list of best AI/ Machine Learning development companies in India and USA are: 1. Xicom Technologies (The Best AI/ML Development Company in India) 2002 Foundation Year: : $25-49/hr Pricing : India, USA, and Dubai Location : 400+ Employees From Chatbots to AI-oriented devices, emerging technologies are setting the new paradigm in the digital era. Besides, from the research lab to businesses, they have become a vital part of any organization or strategy. Learning how to create an AI-oriented app from scratch is a worth-the-value skill nowadays. Xicom Technologies is a well-known with the largest AI platform. The company incepted in 2002, as a one-stop solution for all digital needs and without any doubt is now the market leader in the same field. AI/ML development company in India With the help of Xicom’s Free Project Estimation, it helps a huge number of startups, entrepreneurs, and SME’s lay the firm foundation in the competitive global market. Their Artificial Services in India is known as the bespoke for their quick response and doubt solving capabilities in the region. Indeed, it’s a virtual assistant. Web app development, mobile app development (Android, iOS), Artificial Intelligence, Machine Learning, IoT, Blockchain technology, IT Consulting services, software development, Quality assurance and testing, cross-platform app development, AR/VR. Area Of Expertise: 2. Space-o-Technologies (An Entrusted Name For AI Company In India) : 2010 Foundation Year : $125-149/hr Pricing : India and Russia Location : 250+ Employees Space-o-Technologies is an avant-garde choice among the best . Since, their inception they have been catering service right from a concept to deployment and marketing, and making maintenance the final consideration with the rubric of one of the entrusted Machine Learning services in India. Artificial Intelligence company in India Looking for more? This firm has achieved a leading position in the list of best mobile app development company by following the footsteps of design-led-engineering. Moreover, the Space-O-Technologies professionals have already initiated harnessing the ML features keeping in mind the potential app development. Mobile app development, web app development, app migration services, machine learning, IoT development, customizable solutions Area Of Expertise: 3. Endive Software (A Considerable Choice Among The Top AI companies In India) : 2008 Foundation Year : $99-125/hr Pricing : India & USA Location : 100+ Employees It is a name known for professionalism, dedication, and expertise in the trade of . A worth-the-value option for abeyant business providers. AI/ML company in India So, if you are looking for options that will manifold your business profits and help you drive through digital demurs with utmost satisfaction, Endive Software can be a considerable choice. Their exclusive service to hire Machine Learning developers helps growing businesses with a competitive edge to ace the traits. Counting from customized applications development to blockchain solutions, the company delivers what the client needs. Enterprise solutions, Mobile & web app development, artificial Intelligence solutions, testing and QA. Area Of Expertise: 4. WebNest Software Solutions (Leading Choice For Deep Learning Company In India) : 2012 Foundation Year : $125-149/hr Pricing : India Location : 80+ Employees WebNest Software Solutions has made it to the t with developing applications that stand out in the crowd. With each new dawn and dusk, the company moves towards being the top-most mobile app development company for newcomers in the market. op 20 deep learning services in India Wondering about what makes it a worthy choice? “The Competitive Price“ the company offers is what makes it the right choice for your project successful launch. Their client-first approach is what helps them draw the subtle difference between good and the best AI/ML companies in India. Lastly, their developer's proficiency in creating mobile apps, AI development, Machine Learning, and the never-ending list is worth a look. Mobile app development (iOS, Android), Artificial Intelligence, Blockchain Area Of Expertise: 5. RIT Solutions (One of the best Deep Learning Companies in India ) : 2016 Foundation Year : $60-70/hr Pricing : India Location : 80-100 Employees RIT Solutions is known to provide a competitive edge to businesses already established in the hierarchy. Moreover, it is a digital marketing firm rooted in the region of Gurugram and Jaipur. Their dedication and knocking oneself out approach to deliver tangible, entrusted, and scalable makes them among the top leading deep learning companies in India . In the sea of flooded option, serving to the sheath of numerous clients dwelling from various sectors, the companies strive to be in the list of the top 20 companies in the region. Deep learning company, Native app development, IoT Solutions, mobile and web app development. Area Of Expertise: 6. Technostacks Infotech Pvt Limited (Best Artificial Intelligence Company In India) : 2014 Foundation Year : $80-90/hr Pricing : India Location : 150+ Employees Thinking about what it takes to give an upthrust to your business? Before, it is more important to pay attention to “How”. Quiet simple, align with Technostacks Infotech, an entrusted member of NASSCOM and ISO certified machine learning company in India. It’s a distinctive approach in providing the best service for unique business needs and web-based services across the globe make it a leading choice. Adding on, their developers work hard to develop the latest and exclusive algorithms. The algorithms that work wonders for your business from structured and unstructured data gathered together from different business activities. Machine learning, mobile and web app development, software solutions, cross-platform app development Area Of Expertise: 7. IBM Watson (Among The Best AI Development Companies) : 1992 Foundation Year : $200/hr Pricing : India Location : 2000+ Employees Drive with the latest inventions and innovations in the trade of machine learning technologies, Watson let you ace with the minimum. Whatever, be your requirements, AI technology can be included with the round table discussions by aligning with the surmounted Artificial Intelligence company in India. Nevertheless, you can build models from scratch or even leverage your API’s and customized solutions with keeping your data and business insights to you. Artificial Intelligence, IoT, Customer engagement Area Of Expertise: 8. QBurst (A worth-The-Value Artificial Intelligence Company In India) : 2004 Foundation Year : $80/hr Pricing : India & USA Location : 250-999 Employees QBurst is known to deliver application services and customized consultations for machine learning services in India. Even, it is popular for crafting unique yet reliable mobile and web applications. Their core strength includes in their professionals - Developers, Designers, mobile and web application developers, business analysts, UI/UX developers, and never to forget project management experts. They promise to deliver solutions that lend them a hand to achieve unmountable heights of success. mobile app development, web app development, quality assurance and testing, artificial intelligence Area Of Expertise: 9. Optisol Business Solution (The Best Deep Learning Companies) : 2006 Foundation Year : $90-99/hr Pricing : India and USA Location : 250+ Employees OptiSol Solution is one of the entrusted choices for the leading software development companies with an excellent backdrop in Artificial Intelligence and Machine Learning technology. With their out-of-box thinking approach and inquisitive approach to creating a solution, this firm has crafted a place in the list of leading AI/ML company in India. Their extensive approach offers an industry-specific platform or service with wizards supervision. After all, execution transparency is what matters. Artificial Intelligence, Machine Learning, Enterprise mobility solutions Area Of Expertise: 10. ThoughtSpot (Result-Driven Artificial Intelligence Company In India) : 2012 Foundation Year : $ 100-149/hr Pricing : India and USA Location : 250+ Employees Inching towards ThoughSpot for your project success proves to be a healthy investment keeping in mind the futuristic approach. Their team to hire machine learning developers in India or Artificial Intelligence is here, proffering you with quick and automatic analyzing the data with the right business insights that too with just a touch. What makes the difference? Thought-Spot can align with any reliable on-premise, Big-Data, and cloud platform and providing solutions with a higher proficiency of 85%. Artificial Intelligence, Retail and Ecommerce solutions Area Of Expertise: 11. CloudeDots (In The List Of Best AI Company In India) : 2018 Foundation Year : $90-100/hr Pricing : India and USA Location : 80+ Employees What should be focus points while looking for a reliable company for mobile and web app development? Whatever, be your doubts get it to answer with CLoudeDots professionals, which makes them a reliable choice among the paramount of AI development company in India. Simultaneously, with their roots in deep learning makes their dream turn an operational reality. This firm makes sure to tick all the boxes of your unique business needs with an impeccable user interface, bug-free deployment, and easy to maintainable tools. 12. AbsolutData Analytics (An Excerpt Of Leading Deep Learning Company) : 2001 Foundation Year : $100-149/hr Pricing : India Location : 250 - 999 Employees This leading AI development company in India helps its respective clients to reap profitable decisions to harness scalable heights and manifold business profits. Their experts follow AbsoluteData decision engineering methodology which is the tried and tested approach that helps many companies to use their data in a more streamlined way. Thus, their team of more than 250+ data scientists and engineers makes a pilot project a total customizable AI platform that keeps the potential to include future digital transformation. 13. Good Work Lab (The Top-Most Choice Among The Best AI companies) : 2013 Foundation Year : $125-149/hr Pricing : India and USA Location : 50-249 Employees GoodWork Lab is a well-known, reputed, and popular choice for the latest software and innovative design solutions. Their professionals cater to quality-oriented products and services for clients globally. With it’s innovative and distinctive methodologies, it had crafted a place among the top 20 AI/ML development companies in India. Their experienced professionals and excellent designs make them come out as unique companies in the field. Lastly, their dedicated team ensures to deliver quality, not quantity. 14. Signity Solutions (A Trusted Choice For AI Development Company) : 2009 Foundation Year : $80-100/hr Pricing : India Location : 100+ Employees Your decision of aligning with Signity Solutions is an innovative move for your business. Their approaches and methodologies make them the top-level performer, offering unparalleled AI/ML services in India. With their professionals, you will get an outlook of the latest values and approaches for developing personalized applications, applications with AI, and have more smart business solutions for meeting the unique customer business needs. 15. DigiFutura Technologies (A Top-Notch AI Development Company) : 2014 Foundation Year : $125-149/hr Pricing : India Location : 10-50 Employees DigiFutura is an award-winning web and mobile application development company cementing its footprints in emerging technologies like AI/ML. Being a considerable choice among the leading AI/ML development companies in India, they have marked their presence in the USA and Vietnam. Moreover, DigiFutura has been awarded as one of the Top 10 Best Startup in Mobile Application Development Companies in 2016 by Startup City Magazine and also consists of an elite group of digital agencies with proficients under their sheath, makes them a recognized option for a Mobile App Developer in India by Clutch. 16. Cygnet Infotech (A Trusted Choice For Deep Learning Company) : 2000 Foundation Year : $99-120/hr Pricing : India and USA Location : 200+ Employees Cygnet is a reliable technology partner for your project successful launch with its 18 years of experience. Their developers make sure they provide clients with what they need the most. The company has been delivering extensive digital services to the clients globally under 1400 software service under their hat. Cygnet strives to perfection constantly, with its bespoke expertise in Java technology, Microsoft, PHP, Mobile, PHP, Cloud, Testing, CMS platforms, and rich technologies. Their digital transformation takes the upsurge with the best strategic and consultative approach, perfectly amalgamated with the experiences and knowledge, thoughts and customer support for bracing their identity in the competitive digital world. 17. Ipraxa Inc . (Top Machine Learning Company In India) : 2004 Foundation Year : $80-90/hr Pricing : India and USA Location Employees: 50-249 Begin their journey with a team of few members in 2004, their dedication, direction, and, distinctiveness made them craft a place in the list of leading AI/ML development companies in India. Now, their team consists of 50+ techno-geeks who are ever ready to harness the best in the digital industry, making them a pioneer among the list. Your requirements, their solutions. This approach had made them gain a lot of accomplishments in a small tenure with delivering uttermost client satisfaction. 18. Focaloids Technologies (A Name For Deep Learning Company In India) : 2012 Foundation Year : $70-80/hr Pricing : India Location : 100-250 Employees Are you up for options to modernize your conventional approaches for your business? The address has been changed! Focaloids Technologies is the new hub for every digital challenge encountered in the journey of enterprise applications development. Not confined to this, they have a strong workforce experienced in developing applications that are reliable, operational, and innovative with equal knowledge of machine learning. Being among the top machine learning companies in India, their core focus is to deliver an unforgettable experience without breaking your budget. 19. Prolitus (An Experienced Name For Machine Learning Company In India) : 2005 Foundation Year : $80-90/hr Pricing : India and USA Location : 250-999 Employees If you are looking for dynamic solutions in the vicinity of machine learning companies in India, look no further than Prolitus Technologies. Their experts have been catering to numerous clients across the globe with varied business needs. They dwell with 200+ seasoned techno geeks in their team which makes the company craft it’s placed in the already established hierarchy as an innovator. Moreover, their Artificial Intelligence solutions blended with the state-of-art machine learning tools make it the top-most choice for startups and SME’s. 20. Arstudioz (A Reliable Deep Learning Organization In India) : 2017 Foundation Year : $70-80/hr Pricing : USA and India Location : 100+ Employees Arstudioz is an unparalleled software development company leaving its everlasting impression in the field of deep learning services in India. Their extensive approach of app development that creates user-friendly mobile apps to maximize customer engagement and transition gratifying overall utter satisfaction with your customer. Their in-house team includes savvy experienced professionals for all the famous platforms — iOS and Android, laptops and desktops, connected devices, to emerging technologies like blockchain and IoT. So, if you have a fresh idea, this can be your right destination for turning your idea into a reality. However, the million-dollar names are concerned, such firms help them with their maintenance too. In this blog, I have clutched the trusted AI/ML companies in India that are ready to straddle the fence and leave their first impression making it the last. Wrapping Up: Thus, if you are also the newcomer in the landscape and holds the capacity to leave a mark of your own with the help of the right digital partner. Don’t forget to make the choice among the top. After all, they strived for the place.

Top 20 AI & Machine Learning Companies In USA & India 2019 Edition

Talk to your data! You can view the API documentation here: http://docs.dhignite.com/ What if, instead of running complex SQL scripts, you could simply ask your database a question? What were my sales yesterday? What were my top selling products in June? This natural search capability has become more common over the past year as companies such as ThoughtSpot, Salesforce, and Tableau all develop similar technologies. This trend of “data democratization” is forcing us to leverage existing technology in new ways. The idea that data should be accessible to the average end user, not confined to data analysts or forced to go through a complex queuing system where time and convenience often rule the day, is our new expectation. Data should be accessible beyond the dashboards and beyond analysts. That is why we developed a natural language to SQL API and interface. We utilize the latest in natural language processing and machine learning to convert questions into structured SQL queries. Below you can see that we have utilized the API as part of a chat interface. You can see the user asks a question, the engine contrasts the question against the database metadata and then return an answer. You can also see the SQL that was generated as part of that response. We are now ready to make this API publicly available, so that others can build similar natural search capabilities into their own products. No more forms for generating reports, instead use natural search to empower the user to ask new questions. You can view the API documentation here: http://docs.dhignite.com/ Sending a request for a SQL-query, based on a natural language questions is simple. { "token": "1242ab77d1202aff87bd0b9e96e5a436", "usertoken": ["1242ab77d1202aff87bd0b9e96e5a436"], "database": "foodmart_demo", "params": "total sales" } This request will generate the following response; First it evaluates the “foodmart_demo” database metadata and then produces the SQL response. {"status": "OK","data": {"trash_list": []},"result": " ","method": "server.getSql"} SELECT SUM(sales_fact.store_sales) as `sum.sales_fact.store_sales`\nFROM sales_fact\n User credentials In API v1 we use simple token-based authentication and authorization. To access the Engine API you need the master token and additional user group’s tokens. You will need additional tokens if you want to create several user groups with different access levels. By default, the admin group exists and the admin token is equal to the master token. Each token is a string (32 bytes length), for example “1242ab77d1202aff87bd0b9e96e5a436”. If you are our client, then you already have such a token. Otherwise, use a string “demo” instead of a token. In this case, you will have read-only access to the “foodmart_demo” database configuration only. The Engine Server Endpoint To access the Engine API you need the Engine Server Endpoint URL. For most users it is: https://msg.dhignite.com/api/1.1/nlq 1.1 is the API version number. Database Configurations To respond to a user request, the server needs to have the database metadata. It is JSON-based file. The “demo” user has access to the “foodmart_demo” configuration (usually, another user also has access to this configuration). You need to place the database configuration file at the .Msg Cloud Processing Server (We will do that for you). You can place the finished file or the administrator will create an “empty” file for you, and you will change it using the API or the site. In this manual, all examples will use the “foodmart_demo” database configuration. Demo of API in Action You can sign up on for access to a demo account so that you can utilize the API as part of a chat interface here: http://docs.dhignite.com/demo/ Please reach out directly or through the website for any question about the system.

Introducing the First Natural Language to SQL API

It’s been an exciting, but complex year in the data world. Just as , the data tech ecosystem has continued to “fire on all cylinders”. If nothing else, data is probably even more front and center in 2018, in both business and personal conversations. Some of the reasons, however, have changed. last year On the one hand, data technologies (Big Data, data science, machine learning, AI) continue their march forward, becoming ever more efficient, and also more widely adopted in businesses around the world. It is no accident that one of the key themes in the corporate world in 2018 so far has been “ ”. The term may feel quaint to some (“isn’t that what’s been happening for the last 25 years?”), but it reflects that many of the more traditional industries and companies are now fully engaged into their journey to become truly data-driven. digital transformation On the other hand, a much broader cross-section of the public has become aware of the pitfalls of data. Whether it is through the very public debate over the , the Cambridge Analytica scandal, the massive Equifax data breach, GDPR-related privacy discussions or reports of growing , the data world has started revealing some risks of AI government surveillance in China darker, scarier undertones. Both are the flipside of the same phenomenon, which has been brewing for many years but is now in full display: just about everything (whether personal or professional) is rapidly getting digitized, and data technologies are becoming more adept than ever at processing and analyzing this massive data exhaust, increasingly in real time. From this can result . The debate on how to combine this great power with a necessary sense of responsibility has become essential. both magic and abuse Let’s highlight some of the key trends and events of 2018. Infrastructure & Analytics From an industry standpoint, the data ecosystem remains as exciting and vibrant as ever, with a rich tapestry of innovative startups, mature “scale-ups”, and many aggressive public technology vendors. Most importantly, many customers large and small are those technologies , and reaping undeniable value from their efforts. deploying in production at scale As the cycle of replacing older IT technologies with more modern data products continues, it seems that the Big Data market (infrastructure, analytics) is cycling through the early majority of buyers and transitioning into the late majority of the traditional adoption curve. In addition, the data world continues its inexorable evolution towards the cloud. It is actually staggering to see how fast the large public cloud providers (AWS, Azure, Google Cloud Platform, IBM) are growing, considering they already each generate billions of dollars of revenues every quarter. The trend raises ongoing concerns around , and this may open up opportunities for startups offering solutions. However, to date even companies adopting multi-cloud strategies tend to still rely on one vendor as their primary provider. vendor lock-in multi-cloud As they keep growing, large cloud providers increasingly compete with each other by offering a wide array of Big Data, data engineering and machine learning tools through their platforms (e.g., Amazon Neptune, Google AutoML, etc.) — and often with aggressive pricing, to attract more developers, as their true business model is data storage. As the scope and sophistication of such tools keep growing, this has a big impact on the data technology landscape, making it arguably , at least for broad, horizontal opportunities. A bit more every year, the list of product announcements at big annual cloud vendor conferences (see AWS , for example) sends shockwaves in the startup industry, as they put cloud vendors in direct competition with dozens of VC-backed startups in one fell swoop. It will be interesting to see how public markets react to the upcoming Elastic IPO, an open-source software company that saw Amazon launch a direct competitor, Elasticsearch, three years ago. harder for startups to compete re:Invent Plenty of opportunities for startups remain, however, as long as they are sufficiently differentiated. Many in the space are scaling fast, and there are a number of particularly interesting, fast-growing segments in the infrastructure and analytics part of the ecosystem, including /real-time, , and . The explosion of interest in AI has also led to great opportunities (and a lot of funding) in , , tools, and platforms enabling the streaming data governance data fabrics/virtualization AI chips GPU databases AI devops deployment of data science and machine learning in the enterprise. Machine Learning & AI It’s certainly been a , with anything from the prowess of AlphaZero to the staggering pace of release of new advances — new forms of Generative Adversarial Networks, Vicarious’ new Recursive Cortical Networks, Geoff Hinton’s new Capsule Networks. AI conferences like NIPS have grown to attract 8,000 people and thousands of academic papers are being submitted every day. wild year in the world of AI research At the same time, the pursuit of , perhaps thankfully so. Much of the current wave of excitement (and fear) about AI results from the impressive performance of deep learning since 2012 but, in the AI research community, there’s a as some question the foundations of deep learning (backpropagation) and others look to move past what they consider “brute force” approaches (lots of data, lots of computing power), perhaps in favor of more neuroscience-based approaches. AGI remains elusive growing sense of “what now?” Far from fearing robot world domination, many in the AI research community are concerned that continued over-hyping of the field may eventually disappoint and lead to another AI nuclear winter. Outside of AI research, however, we are just at the beginning of a wave of deployment and application of deep learning in the real world across a variety of problems involving speech recognition, image classification, object recognition and language, in different industries. If the infrastructure and analytics part of the ecosystem is getting to the late majority, we’re still very much in early adopter territory for enterprise and vertical AI applications. The that started a year or two ago has mostly continued unabated, even though the AI startup market is (arguably) showing signs of finally cooling down. Expectations, round sizes and valuations remain high, but we are certainly past the phase where big Internet companies would snap up very early AI startups at high prices just for the talent. The air is also clearing up a bit and revealing “real” AI startups, versus a number of other companies that were leveraging the hype. Some of the AI startups that were founded in the 2014–2016 time frame are starting to hit early scale, and many are offering increasingly interesting products including health, finance, “industry 4.0” and back office automation. Deep learning will continue bringing a lot of value in real world applications for years to come, and vertical-focused AI startups have many great opportunities ahead of them. Cambrian explosion of deep-learning based startups across industries and verticals This continued explosion is very much a global phenomenon, with Canada, France, Germany, the U.K. and Israel being particularly active. However, , with reports of government-led pooling of data at mind-boggling scale (across Internet companies and municipalities), rapid advances in areas such as facial recognition and AI chips, and gigantic rounds of financing for its startups: according to CB Insights, China accounted for only 9% of global AI deal share but nearly 48% of global AI funding in 2017, up from 11% in 2016 (see some examples below). China seems to be playing at a completely different level in AI In the same vein, (and ownership and security) are emerging as a major concern around the world. In the early days of the Internet, data privacy was about protecting what we did online, a comparatively small portion of our activities. Correspondingly, only a small (albeit vocal and passionate) minority of people truly cared. As just about every aspect of our personal and professional lives is now connected to the Internet through an ever increasing array of connected devices, the stakes are changing. With its ability to spot anomalies in massive data sets, predict outcomes and recognize faces, issues of data privacy AI is compounding the data privacy problem. A separate but related concern is that a lot of this data is owned by large Internet companies (GAFA). Some, like Facebook, have proven to be a less than perfect steward of it. Nonetheless, this data provides them an unfair advantage in the race to produce ever more powerful AI. Against those issues, an emerging theme is to think of the , as well as a way for others, outside of GAFA, to produce great AI. Crypto economics are viewed as a way to incentivize individuals to provide their personal data and for machine learning engineers to build models by processing this data anonymously. It all remains very experimental, but some early marketplaces and networks are emerging blockchain as a possible foil against the risks of AI The 2018 Landscape Without further ado, here’s our 2018 landscape. Quite semantic note: buzz terms come and go. Fewer people speak about “Big Data”, many more about “AI”, often to described the same reality. Consequently, we have slightly rebranded our 2018 landscape: it is now called the “Big Data & AI” landscape! To see the landscape at full size, . click here (The image is high-res and should lend itself to zooming in well, including on mobile) To download the full-size image, . click here To view a full list of companies in spreadsheet format, . click here This year, my FirstMark colleague provided immense help with the landscape. Demi Obayomi We’ve detailed some of our methodology in the notes to this post. Thoughts and suggestions welcome — please use the comment section to this post. Who’s in, who’s out On the exit front, the last year (since our 2017 landscape) has seen solid, but not extraordinarily strong. A few key companies appearing on the landscape went public, in particular Cloudera, MongoDB Pivotal and Zuora. Others are preparing to go out at the time of this writing, such as Elastic. Some notable acquisitions also occurred, including in particular Mulesoft (acquired by Salesforce post-IPO, for $6.5B), Flatiron Health (acquired by Roched for $2.1B), Appnexus (acquired by AT&T for $1.6B), Syncsort and Vision Solutions (acquired for $1.2B by Centerbridge Partners), Moat (acquired by Oracle for $850M), Integral Ad Science (acquired by Vista Equity Partners for $850M), eVestment (acquired by NASDAQ for $705M) and Kensho (acquired by S&P Global for $550M). It is worth noting that, other than Mulesoft, all those companies are headquartered on the East Coast (New York, Boston and Atlanta). Many other companies were also acquired for smaller amounts: Gigya (SAP), Blue River Technology (Deere & Co), CoreOS (Red Hat), Guavus (Thales), Lattice Data (Apple), Socrata (Tyler Technologies) and PracticeFusion (AllScripts). On the investment front, this was a year of big financing rounds for some Big Data and AI startups, particularly in China, with a number of oversized investments including Bytedance ($3B in total across 2 rounds in 2017), NIO ($1.6B across two rounds in 2017), and SenseTime ($850M across two around in 2017 and 2018). Major rounds of US companies appearing on the landscape include Snowflake Computing ($263M Series A — see our recent at ), Cohesity ($250M Series D), Dataminr ($221M Series E), Affirm ($200M Series E), Rubrik ($180M Series D), Qualtrics ($180M Series C — see an older but still relevant at Data Driven NYC), Tanium ($180M private equity round), ThoughtSpot ($145M Series D) and Coveo ($100M private equity round) and C3IoT ($100M Series F). fireside chat Data Driven NYC fireside chat ____________________ NOTES: As every year, we couldn’t possibly fit all companies we wanted on the chart. While the general philosophy of the chart is to be as inclusive as possible, we ended up having to be somewhat selective. Our methodology is certainly imperfect, but in a nutshell, here are the main criteria: Everything being equal, we gave priority to companies that have reached some level of market significance. This is a reasonably easy exercise for large tech companies. For growing startups, considering the limited amounts of data available, we often used venture capital financings as a proxy for underlying market traction (again, probably imperfect). So everything else being equal, we tend to feature startups that have raised larger amounts, typically Series A and beyond. Occasionally, we made editorial decisions to include earlier stage startups when we thought they were particularly interesting. On the application front, we gave priority to companies that explicitly leverage Big Data, machine learning and AI as a key component or differentiator of their offering. As discussed in the piece, it is a tricky exercise at a time when companies are increasingly crafting their marketing around an AI message, but we did our best. This year as in previous years, we removed a number of companies. One key reason for removal is that the company was acquired, and not run by the acquirer as an independent company.. In some select cases, we left the acquired company as is in the chart when we felt that the brand would be preserved as a reasonably separate offering from that of the acquiring company. As always, it is inevitable that we inadvertently missed some great companies in the process of putting this chart together. Did we miss yours? Feel free to add thoughts and suggestions in the comments. The chart is in png format, which should preserve overall quality when zooming, including on mobile. As we get a lot of requests every year: feel free to use the chart in books, conferences, presentations, etc — two obvious asks: (i) do not alter/edit the chart and (ii) please provide clear attribution (Matt Turck, Demi Obayomi and FirstMark Capital). Disclaimer: I’m an investor through FirstMark in a number of companies mentioned on this Big Data Landscape, specifically: ActionIQ, Cockroach Labs, Dataiku, Frame.ai, Helium, HyperScience, Kinsa, Timber, Sense360 and x.ai. Other FirstMark portfolio companies mentioned on this chart include Bluecore, Engagio, HowGood, Payoff, Knewton, Insikt, Optimus Ride, and Tubular. I’m a small personal shareholder in Datadog.

Great Power, Great Responsibility: The 2018 Big Data & AI Landscape

It all started when my father let me play with a soldering iron for the first time. Together we made a simple buzzer out of an old relay. Basically, if you apply an AC current to a relay, it will buzz, so we just wrapped it in a plastic box and soldered in two wires. I took it to my school and was showing it to everyone I knew. I was in the second grade. From that moment on, electronics became my hobby. While my friends were playing football outside, I was reading old Soviet books, and trying to make simple circuits work, even though I didn’t understand what exactly was going on. The title says “for the beginner radio-amateur” Analogous electronics was fun, but more complex circuits, like FM transmitters or receivers, rarely worked for me. Calculating and tuning them required solid knowledge of physics, which I didn’t have at that age. No wonder I was really amused by logical elements. It was so much simpler in the discrete world: no resistors or capacitors, just ones, and zeros. Draw a few boxes with different symbols inside, connect them with lines, put it on a board, and bang! You have a super fancy thing with a bunch of LEDs blinking in a specific order. As soon as I learned what the computer was, I immediately started dreaming about having one. I’ve got mine only in 7th grade, and before that, the smartest device I had was an old Soviet programmable calculator. It taught me concepts of , , and a -loop. It also had a decent 5V DC power adapter, which I used in dozens of my little projects, but real computers were so stunning and mesmerizing. GOTO IF WHILE Electronica MK-61 The early 2000s were the era of gaming clubs and “internet cafes” in ex-Soviet Union. I was saving all my pocket money so that I could once or twice a month get an hour of internet surfing at the local post office. I was quickly learning how the internet works, what are the , , and . By the 6th grade, I was already dreaming about having my own homepage. I even had one, created and hosted at one of the WYSIWYG website services, but it didn’t count as I didn’t program it myself. websites email search engines A typical computer club I’ve got my first computer right before starting my 7th year at the school. Single core , , , and a CRT monitor — pretty decent configuration for 2003. My very first CD was a compilation of various pirated (in our area there was no other) web-design software, along with an interactive and tutorial. Pretty soon I officially had a homepage, fully designed and implemented by myself. It was a long clunky thing on a yellow background, full of random text uglified in . I was the happiest kid on the planet! AMD Athlon 2000 XP 512 Mb Hynix RAM 80 GB Seagate Barracuda HDD, Nvidia GeForce 4 17" Samsung 76BDF HTML JavaScript Xara 3D I didn’t know what exactly the word “ ” means, but I knew for sure that my “ ” had to be more “ ,” so pretty soon I’ve got my first book on a real programming language: an introduction into . And then another one, about — a fancy, object-oriented Pascal, created by Borland. dynamic web pages dynamic PHP Delphi My first programming books Early 2000s were also the time of amateur ethernet LANs tangling around every big residential building. I was connected to , a.k.a. , named against our district number. My nickname was (a.k.a. ) or sometimes , or even on ocasions. It was a wild time of unpatched, windows machines, which anyone could remotely reboot by simply typing followed by an in a terminal. I’ve created a simple prank program in Delphi, which I would sneak onto my classmate’s computer, and then remotely open his CD-drive, flip the screen, or send weird sounds through the speakers. I’ve almost reached the state where I could monitor his screen, but suddenly antiviruses got smarter, and all my prank efforts were shut down. Nevertheless, I’ve learned some basic networking. z53 Zone 53 BSOD Blue Screen Of Death [b.s.o.d] -~=[B$0d]=~- pre-service pack one shutdown IP These things you would find on every roof or service floor of any residential building Somewhere around 2005, my school informatics teacher introduced me to the city’s programming club. Kids there were learning Pascal and then competing in solving various algorithmic puzzles taken from informatics olympiads. Best ones were sent to the national olympiad, and from there to the international, etc. It was a place full of interesting people like , or the other guy, whom I introduced to Linux, and the next month he was already building from scratch… in the 5th grade. this guy Gentoo Competing in olympiads taught me some real programming and algorithmic thinking, which helps me out till these days. That was also the place where I started learning English, as we were often practicing on tasks taken from old international contests and also competing remotely in . USACO Competitions were fun, but I was leaning more towards “tangible” programming — making something, that many people can interact with, and feel the result. I’ve created a very basic CMS in PHP. It didn’t have a template engine, and the whole logic was primarily based on overusing the statement. I didn’t have anything in particular to tell the world about, I just wanted to program, so I was simply running a website for someone else on my CMS, and it was about . Every kid was sick about parkour at that time, so the second user of my CMS turned out to be my school friend, who was running a local parkour portal. include gymnastics By 2006–2007 I had a decent skill, so I started taking occasional freelance jobs. Mostly they were simple contracts, but the biggest one was a “ application: we were scrapping news from some Spanish websites, running them through Google translate, and re-publishing in English. It was a savvy way for the customer to game search engine ranking algorithms, and for me, it was a rather complex several months project, that gave me , which I spent on buying an electric guitar. HTML/PHP PSD to HTML serious” first real $200 Two websites that were served by my first CMS (from web.archive.org) I wasn’t particularly good in programming olympiads and never compete at the international level, but my national results allowed me to enter a top university in the capital of Belarus without passing any exams. I’ve chosen with specialization in . University education consisted mostly of hardcore math, to which I developed taste only by the end of a third year, when it was already too late, and less than mediocre programming courses. I’ve gained some useful knowledge about , but mostly, university taught me how to pass exams while putting in the least effort possible. Applied Math Information Security cryptography fundamentals The summer after my second year at the university I spent in the USA. My school friend and I joined a program. It’s a program, where you get a visa for a summer and are supposed to work for three months at an entry-level job somewhere in the USA. The fourth month you can spend on traveling around the country. Our summer destination turned out to be — a b̶e̶a̶u̶t̶i̶f̶u̶l̶ and s̶a̶f̶e̶ coastal casino city in . We were 13 students living in a three bedroom apartment…Most of my roommates were selling ice-cream, washing dishes, or working at construction, but I’ve got my first freelance contract with a Silicon Valley company, which was paying twice as much compared to dishwashing! The only problem was that our house didn’t have a reliable wi-fi. Every morning I had to walk 2 miles to the public library, work there until 6 pm, and then move over to the mall, where I could catch a free wi-fi from a cafe, and work for an extra 2 hours until my laptop battery would drain. I was writing website scrappers for some education startup in . It was mostly a routine job of copying & modifying the same boilerplate code, but there were also gems. Some sites were actively protecting themselves from scrapping, so we were using a sophisticated pipeline, consisted of connecting via , over several machines, and receiving parsing tasks from a . If there’s something that job taught me, it’s a skill, which I nearly perfected. Work & Travel J1 Atlantic City New Jersey Java Selenium-controlled browsers TOR distributed AWS retry-queue regexp My stuff over my bed right before leaving Atlantic City After returning from the USA trip, I’ve teamed up with my university friend , and we started working on freelance contracts together. We were working from his home, and it was taking most of our time so that almost nothing was left to actually studying at the university. We started from small PHP contracts but quickly switched to and single page web applications written in various javascript frameworks. Over time I gravitated towards the backend, and Andrew became a pure frontend developer. By the last year of study, we were deep into freelance work, and we had to choose what are we going to do after graduation: get a “normal” (regular) job, continue freelancing as is, or scale up by hiring more people. Needless to say, we’ve chosen the latter. That’s how the next chapter of my engineering, and now entrepreneurial career began — an outsourcing company . Andrew Python Upsilon IT My desk at the first Upsilon IT office Upsilon had a pretty rough start: we didn’t have enough experience for the jobs we were taking, we didn’t know anything about hiring, and couldn’t hire people we needed, we didn’t know how to deal with customers. We had to overcompensate with immense amounts of work and resourcefulness. A few times we were running into situations when every client delayed payment, and on a payday, we had 0 on our bank account, so we had to borrow money from relatives to pay out salaries. We survived, everything stabilized eventually, and now Upsilon IT is a happy, ~50 people company, .Nothing has taught me as much as Upsilon yet. Besides the engineering skills, it showed me how to deal with the world stuff, not challenges from the university or competitions. I’ve learned how to communicate with people, how to understand what they actually want ( ), how to convey thoughts, how to hire, how to fire. University wasn’t teaching any of these. offering all sorts of web engineering real fake and need Outsourcing was of course interesting and well paying, but we always wanted something bigger — our own product. Oslo, during the trip to the client In 2014 I participated in the , and suddenly won a green card. We had a lot of discussion with my wife, but the conclusion was that if we go — we go to the Silicon Valley, to learn how the real IT industry works, and how people build . US diversity lottery real stuff We moved to the USA in spring 2015. I wanted to try working for a startup as a full-time employee, so I sold my half of Upsilon IT to Andrew, as I wasn’t helping the company anymore. After temporarily settling in Santa Clara I began my job hunt. Big companies never attracted me, so I was never seriously considering them, though I tried my luck with a few. I wanted to observe things from a close distance, and my real goal was a “ ” stage startup with a hundred people or so. late early I started going through interviews, and quickly realized that it wasn’t what I expected! I thought that they would consist mostly of conversations about my experience, but people were asking me ! It’s somehow assumed here that by asking a person and , you are testing for problem-solving skills. But I haven’t done that for years! After failing first few interviews, I spent two or three weeks just brushing up what I was doing back in high school. That quickly paid off — whiteboard interviews became a breeze, I’ve passed an on-site with Google, but even before I knew the result, I’ve already accepted an offer from — a prominent BI startup from Palo Alto. I knew nothing about BI, but the engineering interview was so in-depth, and people were so smart, that I was eager to start working with them immediately. algorithms RMQ LCA ThoughtSpot An after-interview post card signed by the ThoughtSpot engineering team Next 2.5 years I spent in ThoughtSpot. For the first time in my life, I was touching “big engineering.” These people wrote a database from scratch and were shipping their own appliance, and I… I’ve never seen a server unit before. A transition from an employer to an employee was also pretty unusual. I now had a boss, or even two, and was receiving, not distributing tasks. Eventually, I fit in, and the next two and a half years were the most significant boost of my engineering skill. I learned about in-memory databases, highly parallel computations, distributed systems, high availability, enterprise products, etc., etc. I became proficient with Docker and Kubernetes, and even wrote a about them. Visited an OSDI conference, and learned Go programming. ThoughtSpot has recently become a unicorn, but I it almost a year ago to embark on my own journey again. few articles left OneBar team in the Minsk office As I mentioned before, Andrew and I always wanted to build a real software product that would be serving real people’s needs. I always considered my move to the Silicon Valley as a learning trip, not a career destination, and a year ago I started to feel that there’s enough learning — it’s time for action! We picked an area where we had the most experience — communication in large distributed teams. Communication and knowledge sharing is the essence of any company productivity, and yet it wasn’t right at any project, we had to deal with before. We’ve assembled a four people team (of me, Andrew, and two more Upsilon guys), tried out one idea, which didn’t quite work, and right now working on the second one, which we feel has a much higher potential. I’m on my own again, and everything feels new and exciting. I have a lot of enthusiasm for our new project, and I’m again learning! Recently I had to temporarily return to ThoughtSpot as a part-time contractor so that I can support my life here in Silicon Valley, but I’m still dedicating most of my time to the project. I’m picking up lots of new skills, which I would never need, had a been a regular software engineer: , , , , , , and many more. Even though it doesn’t pay me back in money (yet), I’m already considering the whole idea a success, as it’s all essentially a way way better learning experience than any top university. design marketing talking on the phone (!) writing presenting telling stories With that, I’m going to conclude my story for now, and go back to working on . Looking forward to writing the next article about all the learnings we’ve gained in the process of building it! OneBar I encourage you, the brave reader, who’ve reached to the bottom of this article, to go and check out what we’re building , and consider signing up for a beta if your team actively uses Slack. here Thank you!Good luck, and peace! Links Another reason why your Docker containers may be slow Kubernetes for dev infrastructure A Knowledge Base in your Slack

My engineering journey to date

In my last I was talking about Kubernetes and how uses it for its dev infrastructure needs. Today I’d like to follow up on that with a rather short but interesting debugging story that happened recently. It re-iterates on the fact that containerization virtualization and demonstrates how containerized processes can compete for resources even if all cgroup limits are set to reasonable values, and there’s plenty of computing power available on the host machine. blog post ThoughtSpot != So, we were using our internal Kubernetes cluster to , and everything was going great except one thing: when launching Dockerized copies of our product, we saw a much much worse performance than we expected. Each of our containers had generous CPU and memory limits of 5 CPU / 30 Gb RAM set through the Pod configuration. On a virtual machine that would be more than enough for all the queries upon our tiny (10 Kb) test dataset to fly. And on Docker & Kubernetes, we were able to launch only 3–4 copies of a product on a 72 CPU / 512 Gb RAM machine, before things were becoming too slow. Queries that used to finish in a few milliseconds were now taking a second or two, and that was causing all kinds of failures in our CI pipelines. So, we dove into debugging. run a bunch of CI/CD and dev-related workflows The usual suspects were, of course, configuration errors that we might have made while packaging our product in Docker. However, we couldn’t find anything, that might have caused any slowness, compared to VM or bare metal installations. Everything looked correct. As a next step, we ran all kinds of tests from a package. We’ve tested CPU, disk, RAM performance — nothing looked any different from bare metal. Some services in our product save detailed traces of all activities, which can later be used for performance profiling. Usually, if we’re starving on one of the resources (CPU, RAM, disk, network), there would be significant skews in timing for some calls, and that’s how we determine where the slowness comes from. In this case, however, nothing looked wrong. All the timing proportions were the same as in a healthy configuration, except that every single call was significantly slower than on bare metal. Nothing was pointing us in the direction of the actual problem, and we were ready to give up, but then we found this: . Sysbench https://sysdig.com/blog/container-isolation-gone-wrong/ In this article, the author analyzes a similarly mysterious case, where two, supposedly lightweight, processes were killing each other, when running inside Docker on the same machine, even though resource limits were set to very conservative values. Two key takeaways for us were: The root cause of his problem ended up being in the Linux kernel. Due to a kernel dentry cache design, the behavior of one process was making kernel call significantly slower, and this was directly affecting the performance of the other. __d_lookup_loop The author used to track down a kernel bug — a beautiful debugging tool, which we never used before (what a shame!). perf perf (sometimes called perf_events or perf tools, originally Performance Counters for Linux, PCL) is a performance analyzing tool in Linux, available from Linux kernel version 2.6.31. Userspace controlling utility, named perf, is accessed from the command line and provides a number of subcommands; it is capable of statistical profiling of the entire system (both kernel and userland code). It supports hardware performance counters, tracepoints, software performance counters (e.g. hrtimer), and dynamic probes (for example, kprobes or uprobes). In 2012, two IBM engineers recognized perf (along with OProfile) as one of the two most commonly used performance counter profiling tools on Linux So, we thought: why can’t it be something similar in our case? We run hundreds of different processes in our containers and they’re all sharing the same kernel. There must be some bottlenecks! Armed with in both hands, we resumed our debugging and it led us to some interesting findings. perf Below are perf recording of a few tens of seconds of ThoughtSpot running on a healthy (fast) machine (left side) and inside a container (right side). We can immediately notice, that the top 5 calls on the right side are kernel related and the time is mostly spent in the kernel space, while on the left side, most of the time is spent by our own processes operating in the user space. More interestingly, a call that is taking all the time is a . posix_fadvise Programs can use posix_fadvise() to announce an intention to access file data in a specific pattern in the future, thus allowing the kernel to perform appropriate optimizations. It can be used in all kinds of situations, so it doesn’t directly suggest where the problem may be coming from. However, after searching our codebase, I found only one place, which had a potential of being hit by every process in the system: It’s in the third-party logging library called . We use it all over the project, and this particular line is in the — perhaps the most critical path in the whole library. It is called for every “log to file” event and multiple instances of our product might be logging very intensively. A quick look at the source code suggests that the part can be disabled by setting a flag: glog LogFileObject::Write fadvise --drop_log_memory=false if (FLAGS_drop_log_memory) {if (file_length_ >= logging::kPageSize) {// don’t evict the most recent pageuint32 len = file_length_ & ~(logging::kPageSize — 1);posix_fadvise(fileno(file_), 0, len, POSIX_FADV_DONTNEED);}} which we immediately tried and… bingo! What was previously taking up to a few seconds is now down to just (eight!) milliseconds. A little bit of Googling took us to and , which further confirmed that this was indeed the root cause of the slowness. Most probably, it was affecting us even on VMs or bare metal, but because there we had only one copy of each process per machine/kernel, the rate at which they were calling was several times slower, and thus not adding significant overhead. Increasing the number of logging processes by 3–4 times, while letting them all share the same kernel — that’s what caused to become a real bottleneck. 8 https://issues.apache.org/jira/browse/MESOS-920 https://github.com/google/glog/pull/145 fadvise fadvise Conclusions While it’s definitely not a new discovery, most people still don’t keep in mind that in the case of containers, “isolated” processes compete not only for , , , and but also for all kinds of . And, because the kernel is unbelievably complex, inefficiencies may occur in the most unexpected places (like a from ). This by no means concludes that containers are worse or better than traditional virtualization — it’s an excellent tool for its purpose. We should just all be constantly aware of a kernel being a shared resource, and be prepared to debug weird collisions in the kernel space. Furthermore, those collisions are excellent opportunities for intruders to break through the “ ” isolation and create all kinds of covert channels between containers. Finally, is a wonderful tool that can show you what’s all going on in your system and help you debug all kinds of performance issues. If you’re planning to run high-load applications on Docker, you definitely should invest time in learning . CPU RAM disk network kernel resources __d_lookup_loop Sysdig’s article lightweight perf perf Links Kubernetes for dev infrastructure My engineering journey A Knowledge Base for your Slack

Another reason why your Docker containers may be slow

Kubernetes is one of the hottest open-source projects these days. It’s a production-grade container orchestration system, inspired by Google’s own and released into the wild in 2014. Thousands of developers joined the project since then, and now it’s becoming an industry standard for running containerized applications. Kubernetes is designed to run production workloads on a scale, but it’s capable of much more. In this article, I’ll talk about my experience setting up a Kubernetes cluster as a core component of a development infrastructure while working at . Borg ThoughtSpot Context is developing a sophisticated system for large enterprises, which runs on top of another Borg-inspired orchestration system called Orion. It was designed internally, at the times when neither Docker nor Kubernetes were publicly available. ThoughtSpot BI A few things to know about ThoughtSpot, which are relevant to this article are: The system consists of a few dozens of services and the overhead of running them all is quite massive. Idling system with very little data requires of RAM, cores and to start up. 20–30Gb 4 CPU 2–3 minutes ThoughtSpot sells its own appliance, typically with at least RAM per-cluster, so that 20–30Gb overhead is not a problem for the product. However, it’s quite an issue for the dev infrastructure. 1TB There’s a lot of retired hardware available for developers in the office. Motivation I was initially assigned to solve an easy-sounding problem: make integration tests faster. There were a few hundreds of Selenium-based workflows, which were running sequentially and taking up to 10 hours to complete. The obvious solution was to parallelize them. The problem was that they were not designed to run concurrently and hence we had to either refactor all tests or provide an isolated copy of the ThoughtSpot system ( ) for every thread to run on. Redesigning tests might look like a cleaner solution, but it would require a tremendous effort from the whole engineering team and a lot of test-related changes in the product, so it was not feasible. We’ve decided to take the second approach, and that left me with the task, I’ve ended up solving with the help of Docker and Kubernetes: a test backend make it possible to quickly (in 2–3 minutes) spin up dozens of test backends with pre-loaded test data, run tests, tear them down, repeat. The path With this task in mind, I started looking for options. Actually, some infrastructure was already in place: we had a VMware cluster running on four servers. Current integration tests were already using it for provisioning test backends, but there were problems: It could only sustain about VMs, after that we would have to buy more of the expensive proprietary hardware. It was already utilized for about 80% by other workflows in the company. one hundred Cloning 10 or more VMs in parallel was blowing up the IO. It would have to move around ~ of disk snapshots, and it was taking forever. 500Gb VMs were taking way more than 2–3 minutes to start up. Virtualization wasn’t a viable option for us, so we turned our heads to containers. In early 2016 we were looking at two main options: and . Docker was an already recognized leader, and LXD 2.0 was only going to be released along with Ubuntu 16.04. However, Docker has a strong bias towards small, single-process containers, which have to talk to each other over the network and form a complete system in this way. LXD, on the other hand, offered something, that looked more like familiar VMs, with the system and the ability to run multiple services inside a single container. With Docker, we had to either compromise on the cleanliness and use in an “LXD way” or re-factor the whole system to make it run on top of Docker, which was not feasible. On the other hand, with LXD we could not rely on the exhaustive community knowledge, as well as the documentation, that Docker had. Still, we’ve decided to give it a shot. LXC/LXD Docker init LXC/LXD I took four machines, each with RAM, cores, 2 SSDs and 4 HDDs, installed LXD and configured a ZFS pool on each node. I’ve then set up a Jenkins job, that would build the project, install it inside the LXD container on one of these machines, export an image and push it to the other three nodes. Each integration test job would then just do , run the tests and destroy containers once done. Because of the nature of ZFS, clone operation was now instantaneous. Each node was able to handle about ten test backends until things would start crashing. This was a great result, much better than what VMware was giving us, but with a major drawback: it wasn’t flexible nor scalable. Each test job would need to know exactly on which of LXD nodes to create its backends and, if it required more than of them, they just wouldn’t fit. In other words, without the orchestration system, it was not scalable. With LXD, at that time, we had only two options: use OpenStack or write our own scheduler (which we didn’t want to write). 256Gb 40 CPU lxd clone current-master-snapshot <backend-$i> copy-on-write 10 , but in 2016 it was all very fresh, barely documented and barely working. I’ve spent about a week trying to configure an OpenStack cluster and then gave up. Luckily, we had another unexplored path: . OpenStack supports LXD as a compute backend Docker and Kubernetes Docker & Kubernetes After the first pass over documentation, it was clear that neither Docker nor Kubernetes philosophy fit our use case. Docker explicitly said that “ ”, and Kubernetes was designed for running one (or few) application, consisted of many small containerized services, rather than many fat single-container apps. On the other hand, we felt that the movement behind Kubernetes was powerful. It’s a top-tier open-source product with an active community, and it can (should) eventually replace our own, home-grown, orchestration system in the product. So, all the knowledge that we acquire while fitting Kubernetes for the dev infrastructure needs we can reuse later when migrating the main product to Kubernetes. With that in mind, we dove into building the new infrastructure. Containers are not VMs We couldn’t get rid of the Systemd dependency in our product, so we’ve ended up packaging everything into a CentOS 7 based container with the Systemd as a top-level process. Here’s the base image that worked for us. We’ve made a very heavy Docker image ( initially, after some optimizations), which encapsulates Orion (ThoughtSpots own container engine), which then runs 20+ of ThoughtSpot services in cgroup containers, and that all roughly corresponds to a single node production setup. It was cumbersome, but it was the quickest way from nothing to something usable. Dockerfile 20Gb 5 After that, I took a few other physical machines and created our first Kubernetes cluster on them. Among all of the Kubernetes abstractions, only was relevant to our problem, as it’s really just a container running somewhere. For most of our test cases, we would need to create multiple and having the ability to group them by workload would be helpful. Perhaps are better suited for this purpose, but we’ve decided to exploit a . is an abstraction that would create a number of (according to a ), make sure they are always alive and, on the other end, receive traffic from a and redistribute it across the . assumes that every is equal and stateless so that every new connection can be routed to a random . In our case, we did not create a and just used as a way to group and make sure they get automatically re-created if anything dies. Every test job would then create a for itself and just use the underlying directly. Pod Pods labels ReplicationController ReplicationController Pods Replication Factor Service Pods ReplicationController Pod Service Pod Service ReplicationController Pods ReplicationController Pods Pod networking hack We rely on behaving as real VMs API-wise. In particular, we needed SSH access to every and an ability to talk to dynamically allocated ports. Also, every was obviously stateful, as the image incapsulated the state store in it. This effectively meant that instead of using and load balancing through , we had to break into the directly. We’ve done that by enabling on the Kubernetes master node (turning it into a router) and re-configuring all office routers to route (our ) through the Kubernetes master node. This is a hack which should never be done in production environments, but it allowed us to quick-start the dev infrastructure, solve the immediate problem and start looking into ways how to make our product Kubernetes-ready in the future. Pods Pod Pod Services kube-proxy pod-network ip forwarding 172.18.128.0/16 pod-network terrible Fast forward almost two years and here’s how ThoughtSpot’s dev infrastructure looks right now: Kubernetes cluster is running on 20 physical machines, providing of RAM and cores combined. 7 Tb 928 CPU Every host node is running CentOS 7 with 4.4-lt Linux kernel. We use as an overlay network, and the routing hack is still in place. Weave We run an in-house Docker registry, to which CI pipeline uploads a product image every time the master or release branch build succeeds. We use plugin to provision Jenkins slaves on Kubernetes dynamically. Jenkins Kubernetes We’ve recently deployed on a few nodes and started experimenting with persistent stateful services. and were the essential tutorials. Glusterfs This this MAAS During this project, we’ve discovered another great open-source tool, which helped us a lot with managing physical hardware. It’s called and translates as “ .”It’s a tool, which leverages and to allow dynamic node re-imaging with an arbitrary OS image. On the user side, it provides a REST API and a nice UI, so that you can provision physical machines in AWS style, without actually touching the hardware. It requires some effort to set it up initially, but after it’s there, the whole physical infrastructure becomes almost as flexible as the cloud.Right now we provision plain CentOS 7 nodes through MAAS and then run an script, which upgrades the kernel, installs all the additional software and adds the node to a Kubernetes cluster. ( ) MAAS Metal as a Service PXE booting remote node control Ansible link to a gist Nebula Most of the developers or CI jobs do not interact with MAAS or Kubernetes directly. We have another custom layer on top of that, which aggregates all available resources together and provides a single API and UI for allocating them. It’s called Nebula, and it can create and destroy test backends on Kubernetes, as well as on the old VMware infrastructure, AWS, or physical hardware (through MAAS). It also implements the concept of a : every resource provisioned is assigned to a person or a CI job for a certain time. When the lease expires, the resource is automatically reclaimed or cleaned up. lease LXCFS By default, Docker mounts filesystem from the host and hence (meminfo, cpuinfo, etc) do not reflect container-specific information. Especially, they do not reflect any resource quotas set on cgroup. Some processes in our product and CI pipeline check for the total RAM available and allocate its own memory accordingly. If the process doesn’t check the limit from cgroup, it could easily allocate more memory than allowed by a container quota, and then get killed by the OOM killer. In particular, this was happening with a lot the JS uglifier, which we were running as part of the product build. The problem is described and discussed , and one of the solutions for it is to use . /proc/ /proc/stat here LXCFS LXCFS is a small FUSE filesystem written with the intention of making Linux containers feel more like a virtual machine. It started as a side-project of LXC but is useable by any runtime. LXCFS will take care that the information provided by crucial files in procfs such as: are container aware such that the values displayed (e.g. in /proc/uptime) really reflect how long the container is running and not how long the host is running. /proc/cpuinfo   /proc/diskstats   /proc/meminfo   /proc/stat   /proc/swaps   /proc/uptime Conclusion It took us quite a lot of time to figure all the things out. There was a tremendous lack of documentation and community knowledge at the beginning when we were just starting with Kubernetes 1.4. We were scraping the particles of information from all over the web and learning by debugging. We’ve also made dozens of changes to our product, re-designed the CI pipeline and tried many other things which are not mentioned in the article. In the end, however, it all played out well, and Kubernetes became a cornerstone of dev infrastructure in ThoughtSpot, providing much needed flexibility and allowing to utilize all the existing hardware, available in the office. I left the company in September, but the project got handed over to other developers and keeps evolving. I know that many people are trying to build something similar for their companies, so I would be happy to answer any questions in the comments below. Links A story of debugging a Docker container slowness My engineering journey A Knowledge Base for your Slack

Kubernetes for dev infrastructure

Experienced entrepreneurs know that building successful products is not about damn luck or revolutionary ideas; it’s more about data-driven decisions and careful analysis of the market. There are plenty of books and posts written on this topic, but it still takes days of research to create some sort of a mindmap and understand startup metrics and their common use cases. This post is a quick cheatsheet for entrepreneurs to determine what metrics may be useful to their business. Let’s assume that we are building a social network, and our business model is ad-based. The longer the average session duration in our app, the better, right? It means that our ads will get more impressions from our app users. But what if we are building not a social network but a search engine? Is it a positive change when the average session duration grows? Or it means that our UX is getting worse, and users have to spend more time looking for the requested information? And lastly, what does the average session duration mean for a SaaS application like GitHub or Slack? The problem here is that startups try to create something unique by definition and cannot use the same metrics as a measure of their success/progress. Sometimes, a metric has opposite meanings for different business models. Before We Get to Actual Metrics In order to provide you with any recommendations, we need to find similarities in startup businesses and group them up somehow. I believe that the two main factors that define what metrics are more important for your business are: Business Model All business efforts are always focused on the only goal: maximizing profits. A business model defines revenue sources and all the financial metrics that depend on them. Current Stage In the beginning, every startup faces an atmosphere of extreme uncertainty. As it progresses over time, it validates business hypotheses one by one and picks more and more precise metrics to rely on. To read further about metrics, make sure you know about different , and can define yours. business models and startup stages In terms of this post, we don’t discuss the business models that assume uneven revenue distribution across customers. If your business provides a service to one or two big clients that generate 80% of your revenue, data analysis won’t be as helpful as for the business models above. Note: Absolute Values vs. Growth over Time (MoM, QoQ, YoY) It’s important to understand that you can track both of time when it makes sense. When you need to see how a particular metric grows over time, you can use , , or metrics. absolute values of a metric, as well as its change compared to the previous period Month-over-Month (MoM) Quarter-over-Quarter (QoQ) Year-over-Year (YoY) The most common use of this type of metric is Revenue Growth, which is featured in the table below, but you can apply it to any other metric when it makes sense. To give you an example, let’s compare Revenue Growth of a company below. Total Revenue vs. Monthly Revenue vs. Month-over-Month Here is the graph of its Total Revenue: Looks good, right? Seems like the company is growing. But wait a minute, this is the Graph, which adds up new revenue on top of what they already earned. To be honest, the and is used pretty often to impress prospective investors. Let’s review the graph of the same company: Total Revenue Total Revenue graph doesn’t make a lot of sense Monthly Revenue Monthly Revenue vs. Month Seems like it’s not that good. What experienced entrepreneurs and investors pay more attention to is the Revenue Growth graph, and for this company, it looks like below: Month-over-Month (MoM) Revenue Growth vs. Month This Revenue Growth graph clearly shows that . Moreover, it started shrinking. This may indicate that its business model doesn’t scale well. MoM the company stopped growing North Star Metric Tracking a whole bunch of different metrics is difficult, especially when each one of them has its own priority, and your team is growing quickly. That’s why many businesses try and call it . Here are some notes about using NSM based on our experience: to measure their success/growth using just one metric North Star Metric (NSM) Having NSM in place makes the decision-making process much easier. Ex: If a new feature doesn’t drive NSM growth, it means that your team should stop working on that. It’s pretty difficult to define a single metric that would capture your business goal. That’s why you need to readdress your NSM once in a while, especially when shifting your business priorities. NSM can be a complex metric (ex., custom formula), and almost always, it’s different from business to business. Sometimes companies pick a basic metric as their NSM so their team can understand it easily. Here are some good NSM examples: If you’d like to learn more, have a look at an example of how . Thoughtbot calculates its NSM Financial Metrics This is the type of metric that indicates . Usually, financial metrics are not actionable but show your business's overall success/potential. where your business currently stands Acquisition Metrics These are the type of metrics that indicate . With the help of these metrics, you can project your financial KPIs. how fast your business grows It’s important to note that all the acquisition metrics are usually tracked against different acquisition channels (blended, organic, paid marketing, etc.) to determine the most effective ones. Engagement Metrics If acquisition metrics show how fast you grow, then engagement metrics indicate and the value you bring to your customers. Sooner or later, every business starts to focus on these metrics, especially in a competitive market. the quality of your growth These metrics can be a good start for measuring your startup success, growth, and value. Choose your metrics wisely and reconsider them every time you change the strategy or move to the next stage. While you grow and gain more experience, you will create the whole economy for your product with built-in custom metrics. What’s next When performing a user engagement analysis, every business is trying to define their – an action or a sequence of actions that, with high probability, will convert a user into a long-time customer. Having a defined sticky criterion, you can focus on your marketing efforts and rebuild the UX of your product to inspire your users to follow that lead. “sticky criteria” In our next post, we’ll describe how works and how it helps define your sticky criteria and make user engagement forecasts. Cohort Analysis Also published here .

Startup Metrics: A Comprehensive KPI Cheatsheet for Founders

Last week I tried pushing a new branch to my remote repository and encountered an error message, saying that a certain file I previously committed was too large and exceeded GitHub Enterprise’s file size limit, which is 100MB. I work on an API based system, which handles CSV files, filled with rows of data, as a part of its requirements. We encountered a file which led the system to behave unusually, and therefore decided to test all flows to track down the root cause of this unwanted behaviour. As a part of testing the system I added the file to the project. After that I made some unrelated changes, committed them and moved on. After some progress I tried to push to the remote repo which then led to this error message: I then realised that the file I was testing made its way to my commit, which now prevented me from pushing my local branch to my remote repository branch. Things I tried Here are the solution approaches my team and I went through before successfully pushing: Branch out and delete the file First thing I did was to branch out in order to experiment on it. After that I went on to safely delete the file and try committing. Same error occurred. My first thought was that there was an issue with the commit, maybe a general synchronization issue within my branch which might’ve lead to git not recognizing the file removal. So I decided to revert my changes. After that there were some attempts to remove the file from both the command line and file system, commit, encounter the same problem and revert. Develop from scratch I realized pretty soon this wasn’t the right approach. My next idea was to start developing the feature from scratch, open a whole new branch from dev which wouldn’t include this file to begin with and copy all changes to this new branch. The problem with this approach was that my feature included changing many files, had plenty of dependencies and I didn’t want to risk missing out on any of the changes I made. Revert the specific commit The next thing I tried was going over the log to find the specific commit where the file was initially added and revert this specific commit. Once I committed the revert commit I thought this would solve the issue, but it didn’t. As reverting the deletion of the large file itself didn’t work, I tried doing so by deleting the file’s content. My thought was that maybe there was a problem with deleting the file itself, but maybe committing a change that would reduce the file size would fix this and get rid of the file size limit error. What eventually worked The main conclusion was that these options didn’t comply with the way git handles commits in a branch. When I removed the file and committed the change, the file was gone and no longer tracked. But the commit history is there, which means if I had included the large file in any of the previous commits — it’s not gone. The fact that git stores the commit history is amazing, especially when I need to restore changes or recover data that hasn’t been recorded in a snapshot yet. But, the idea that files in commits are frozen forever is crucial to understand, especially when what we’re trying to do is get rid of one particular file which exists somewhere in one of the commits. Rebase and squash commits The next approach was rebasing. Since we figured out the file exists in one of the previous commits, we decided to rebase the branch by moving back HEAD to the point right before the commit we wanted to edit. The idea was to reduce all the commits we’ve made and squash them all together, in order to fix the large file’s existence in any one or more of these commits. We then moved on to viewing all the commits by running git log, used the commit message to identify the hash for the specific commit we needed to edit in order to move back our branch’s HEAD to the point right before this commit, by using (where X is how many commits you need to move your HEAD up, compared to where it is now). git rebase -i HEAD~X It’s important to use the default action “pick” on the commit we want to keep out of all the existing commits. In this case it would reapply the commit as is, no changes in its contents or message. We used “squash” for all other commits, in order to meld each commit into the previous one. Conclusion Trying to push a large file and encounter a size limit error encapsulates much more than just the file size. In this case it’s understanding how git handles commits in a branch. The way we solved this issue was by understanding that a single commit along the way had the file in it, so we needed to exclude this commit in the chain of commits. Rebase and squash all commits that include the file into a single commit which doesn’t, is our way of recreating history where this file is no longer a part of. Further reading: 1. https://about.gitlab.com/blog/2018/06/07/keeping-git-commit-history-clean/ 2. https://thoughtbot.com/blog/git-interactive-rebase-squash-amend-rewriting-history 3. https://stackoverflow.com/questions/22053757/checkout-another-branch-when-there-are-uncommitted-changes-on-the-current-branch Image credit: Cloud photo created by kjpargeter — www.freepik.com

How To Push To Git When Your File Is Too Large?

I founded to help every developer advance their career. Raise.dev It started with a pilot that kicked off last year. The goal was to help early-career developers overcome the experience gap and get their first jobs in industry. I'm proud to report that the results have been outstanding: 100% job offer rate (highlights: DigitalOcean, thoughtbot, American Airlines, theSkimm) $53k/year average increase to base salary (equity, health insurance, perks excluded) Six months average time to job offer (from three to ten months) The developers came from a wide range of backgrounds: 100% from underrepresented groups in technology Education: From no college to masters degree Gender: Majority identify as women Age: 20s to 40s Parents Transitioning from established careers Today, we're opening our doors for early-access and inviting you to to advance your developer career. join the network The Experience Gap Breaking into a career as a software developer is a classic chicken-and-egg problem: it's next to impossible to get a job without relevant work experience and it's equally, if not more difficult, to get the necessary experience without a job. There's no doubt that the barrier to entry for programming has drastically reduced in the last decade. Software development is more accessible as a career in large part thanks to the deluge of free and inexpensive online resources and the rise in popularity of trade school style coding bootcamps. Even so, early-career developers still face the paradoxical "experience gap" when applying for their first jobs. Studying Didn't Help I've heard repeatedly from developers struggling to get their first job due to a lack of experience and I personally experienced the same challenge in my career. When I was starting out, I applied to close to 30 software developer roles and I was consistently turned away due to a lack of experience. So, the question is, I've spent many years trying to answer this question in service of my own career. "how do I get the skills I need to advance my career as a developer?" My first crack at it was to go the academic route. I racked up a mountain of debt studying computer science at a private university. The experience was somewhat valuable, but school didn't prepare me for a job in the industry. I also tried to teach myself. I read books and tutorials. I worked on side projects. Progress was unbearably slow. I'd spend hours and sometimes days or weeks at an impasse because I lacked some fundamental understanding that was necessary to solve my problem. Solving Real Problems with People My biggest leaps forward as a developer always came from working with people to solve real problems. A few years before I got into the industry I was lucky enough to make friends who were more experienced developers. They helped me get into Linux and build some small web applications with PHP. Most people aren't so lucky to have ready access to experienced people who are willing to help. The biggest step change in my ability as a developer came from an experience with a coworker. I needed to upgrade an open source web framework on a widely-used production application. It was uncharted territory for me and incredibly daunting. Any misstep could cause a major outage. Someone that I looked up to volunteered to pair program with me on the project. It was the first time I had ever pair programmed. They helped me write automated tests and make the necessary changes one by one, each time fully verifying that there were no adverse side effects. We managed to complete the upgrade, but what stuck with me were these new techniques that completely changed how I program. They're commonplace to experienced developers. While you can learn about pair programming and testing in school, the techniques don't really sink in and become useful until you've been able to apply them in the real world. That pairing session was the most impactful learning experience of my developer career. It gives me confidence that the best way to learn is by doing the work with guidance from an experienced collaborator. The Obvious Solution: Give People Jobs I've spent my career at the intersection of software development and education. I've worked in ed-tech, helped start an ,  marketed , and given millions of students . online university APIs by teaching free access to real-world developer tools The problem of the experience gap had been troubling me for some time. I put care and thought into how to help early-career developers overcome the seemingly insurmountable hurdle. I came to a simple, albeit impractical, conclusion: just give people jobs. If they have a job they will get experience and, in turn, will become more hireable. Last year, I decided to see what would happen if I did just that. I announced that I was hiring for a Software Developer Apprentice and within a week I had more than 500 qualified candidates. on Twitter To be qualified didn't require experience, but it did require knowing how to code, being highly motivated, and having a demonstrated need for the opportunity. At first I was surprised, but it seems obvious now that there would be essentially infinite demand for such an opportunity. I originally intended to work with one person, but there were so many deserving candidates that I ended up selecting a small group. I found that the time I spent with the apprentices was both rewarding and impactful. Giving early-career developers real projects and providing hands-on support gave them the experience they needed to get hired by some great companies. All of the apprentices who participated in the pilot received job offers for full-time developer roles from great companies including DigitalOcean, thoughtbot, American Airlines, and theSkimm. Connecting Candidates, Coaches, and Project It's not practical for us to hire every developer candidate – that would never scale. Instead, we been hard at work building a platform to connect developer Candidates with expert Coaches to work on real-world Projects. The platform involves three major components. Candidates, Coaches, and Projects: Candidates get exposure to the tools and techniques that are essential for a career in industry. Coaches give back, learn valuable leadership skills, and also solidify their skills through pair programming, code review, teaching, and mentoring. Projects involve helping non-profits, contributing to open source, and doing real work with engineering teams from our hiring partners. Early-access for Candidates Begins Today Following on from the success from the pilot, we're opening the doors to the platform to developer Candidates for early-access. Starting today, you can to be matched with a Coach who will help you create and execute a personal project-based learning plan. Raise.dev is free during early-access. sign up We're excited to help you advance your careeer. . Meet your Coach Also published on: https://raise.dev/blog/bridging-the-developer-experience-gap

How to Bridge the Developer Experience Gap with raise.dev

I tried to learn Vim, believe me, many times. As many times as I tried, I failed. That is until two weeks ago, when something inside my ancient and slightly decrepit mind clicked, and I started to see the way. I began to, as some might say, Vim. It was very exciting (and yes, I’m allowed to use it as a verb). I don’t have to tell you why you might like to use Vim and I also don’t need to tell you that you need a list of any of the excellent resources out there to learn it. There are as many different approaches as there are key bindings, but I hope to share with you my approach, how it was helpful, and how it might help you. The Past And Why It Didn’t Work Previously, I ran through Vim Tutor, watched some of the excellent talks from Ben Orenstein and Chris Hunt, and played some of the way through , but the experience was esoteric and academic. Vim Adventures On top of that, I made the mistake of installing Thoughtbot’s dotfiles repo which gave me A LOT of options and plugins, adding to the confusion. Not only was I trying to learn Vim, I was also trying to learn twenty plugins simultaneously. They had an amazing selection of tools, but as a starting point, it was too much for me to handle. Furthermore, every time I tried applying some of these fledgling skills to a code base I cared about, the frustration of knowing the code I wanted to write and being unable to manipulate the editor the way I wanted was so maddening, I gave up and went back to Sublime (which I still love). How It Finally Happened First, I found a great tool, , and started playing with it. Despite occasionally getting out of sync, it’s a flash-card style semi-game that helped mebuild some muscle memory. Going through Vim Adventures was supposed to do the same thing, but I put on some good techno music, played Vim Genius, and felt cool when I made progress! It was fun to zone out to and repeat until I got it. Vim Genius Second, I found a great .vimrc. As I mentioned, Thoughtbot was jammed full of plugins, extensions, and remappings. Pairing things down was really helpful and made the whole endeavour more approachable. Luckily, two awesome people at , Cody and Claire, had a good .vimrc that was forked off of the Pivotal .vimrc and could serve as a starting point as well as a curated suggestion of what plugins to start with. You can get it from . Stride our repo There’s also a nice Readme that will help you get all the plugins installed. After that, you should be good to go. I did not further installation nor customization of my vim setup, which is nice. Third, and perhaps the most crucial step, I gave myself missions with specific requirements. I picked tasks that were separate from code I wanted to work in, navigate, change quickly, and separate from code that had time constraints or expectations. Giving myself small things made it achievable and motivated me to see what I could get up to. By removing it from the constraints of performance made it easy to go slow, make mistakes, and look up a hundred different things every few minutes. Reading Code I love reading code. Beyond learning something about code, design, and different technologies, it also makes a great task for using the basic Vim commands for navigating. Beyond the basic up/down/left/right, it’s great to use G, gg to jump to the end and beginning of the file, respectively, $ for the end of lines, and { for moving up and down paragraphs in a Readme. It’s also a great way to start jumping around between files using nerdtree. Simply opening and closing the tree with | and || gives an easy way to jump more globally within a project. This reinforces the muscle memory of the basic commands from Vim Genius but also helps you feel confident as you start looking for something you’ll be able to find it and not get lost! Writing Markdown Writing code is complicated. Trying to write code in Vim while not knowing Vim is even more complicated. It was one of the key points that contributed to my repeated failures to learn. I decided to make my life easier and focus on writing some documents in Markdown. That was more manageable and something I could be successful with. First order of business was Markdown Preview. I loved being able to see how nice my rendered markdown looked. But so much how? I found which was nice and made me feel fancy that I could install a Vim plugin. The instructions on the Readme worked and, word to the wise, if you’re using one of the .vimrc’s mentioned above, you will already have filetype plugin on. vim-instant-markdown But spelling! :set spell. That was nice! And z= over one of my highlighted misspellings gave me the suggested correction. Thanks Vim! And when I was done with misspellings? :set nospell. Ahhhh. I liked line wrapping at 80 characters. Simple enough: set textwidth=80 and if I wanted to reformat, enter visual mode, v, use j or k to select up or down and then gq to reformat. Nice! o and O and a and A and i became reflexive for inserting. Also, zR to unfold your markdown file if it gets folded. The R unfolds all folds, which is what I liked to do when things were folded. And finally, my favorite trick, shift-j to join the line below to your current line! This was when I really started having fun, in part because I like writing things, but also because I could feel the vimness happening. Jumping around, inserting, deleting, creating a structured document quickly, efficiently, and ‘at the speed of thought’ started to make me full of vim and vigor giving me the courage to go further. Further, doing this without a deadline really opened up the opportunity to dig deep and figure out how to use the strange powers of Vim to do what I wanted it to. Coding Ok! Now we can get around a little and sort of write things, maybe now it’s time to write a little piece of code. Start with something small, something like your favorite React tutorial, or maybe a code base you know very well, but something that’s going to require you to navigate the code tree as well as write something. <,f> where <,> is the remapped leader (see your .vimrc) and the f is for CtrlP fuzzy find. Nerdtree is nice for seeing things and getting a lay of the land. But for jumping around quickly and not worrying about your buffers, CtrlP is awesome. I think of it as leader-find and have found it to be the single most important thing for moving around a project when I’m editing, because I’m likely to know where I want to go and don’t need the context of the surrounding files from Nerdtree. Next, copy and paste. I’m definitely still learning this but yy for yanking things out and p for pasting them in is a good, nondestructive place to start. Finally, for when you have “blurgh” and want to change it to ‘blurgh’: cs”’, meaning change surrounding double quotes to single quotes. Read more about vim-surround . here Resources and What’s Next As always, we stand on the shoulders of giants. Here are some of the posts I used as I was learning: https://www.linux.com/learn/using-spell-checking-vim https://robots.thoughtbot.com/vim-spell-checking https://robots.thoughtbot.com/wrap-existing-text-at-80-characters-in-vim https://github.com/suan/vim-instant-markdown https://github.com/tpope/vim-surround I am by no means an expert and haven’t even fully immersed myself into the trough of despair, which I expect to be horrible. I’m excited I’ve broken through that initial barrier, and hope with practice, some compassion from my fellow developers, and a little blessing of the Vim gods, I’ll become more and more proficient in my Vimness. If you are learning it now or want to, I hope some of this has been helpful to get you through the initial barriers. Finally, as with all tech and tools, it is worth noting there is no one true way to learn, and beware of the suggestion that there is a ‘real’ way to learn Vim. Learn it in a way that works for you. If you have any suggestions or if you think I missed anything, please let me know! Senior, Lead, or Principal developer in NYC? ! Want to level up your tech team? ! Stride is hiring See how we do it www.stridenyc.com Originally posted on the Stride Blog . Author: Nichol Alexander

Breaking Through The Vim Learning Curve

A look in to the future 🔮 A lot of time you need to make a system that can send out to yourself or other users , specifically events related to changes in your database. and make this ridiculously easy. notifications based on events Postgres Phoenix By the end of this post you will have: short Setup a Phoenix 1.3 project Setup a broadcast function and trigger in Postgres Used a GenServer to listen for broadcasts from Postgres Used Bamboo to send emails to yourself when users signs up or changes their payment plan. Getting Setup Before you start Please make sure you have the following installed and/or ready to go: Elixir Phoenix PostgreSQL A valid account SendGrid Initialize the project Lets create a new Phoenix 1.3 project called pub_sub_demo mix phx.new pub_sub_demo cd pub_sub_demo mix ecto.create Add Two Additional Dependencies Next we want to add to help with decoding strings sent from the database and to help us send emails. HTTPoison Bamboo To do this in your file add the following to your : mix.exs deps {:httpoison, "~> 1.0"},{:bamboo, "~> 0.8"} Also update the section to include extra_applications :bamboo extra_applications: [:logger, :runtime_tools, :bamboo] Create a model Finally lets create the model we will be using to show off all of this broadcasting goodness. User mix phx.gen.context Accounts User users name:string payment_plan:string Note: We created the _User_ in a new context called _Accounts_ . If you are unfamiliar with contexts they are nothing to be afraid of just a convention phoenix uses to group functionality. They are not special, just a way to bag functions together. Kick it off! Migrate the database, grab your dependencies and lets go! mix ecto.migrate Note: You should now be able to navigate to _localhost:4000_ and see your application running. Though for the purposes of this demo you wont really be viewing any screens. Broadcast Changes with Postgres The whole idea of this is for Postgres to let know when things have changed. To do so we need to set up two things: US A function that takes an action ( , , ) performed on a row for the purpose of broadcasting it outside of the database. INSERT UPDATE DELETE A trigger that calls this function when an action has occurred on a specific table. Create the Ecto Migration mix ecto.gen.migration broadcast_users_table_changes Add the Function and Trigger Update the migration file you created above to include the following If you want to better understand what is occurring here check out this blog post by @kaisersly which largely is the inspiration for what you are reading here: https://medium.com/@kaisersly/postgrex-notifications-759574f5796e Listen for Changes Now that we have our database broadcasting changes that occur on our table we need to be able to listen and act on the same channel. Thankfully Postgrex provides this by default using . users Postgrex.Notifications To use this we must create a responsible for listening to and acting on the messages broadcast. GenServer Postgrex Creating our GenServer Create the file and populate it with the following: lib/pub_sub_demo/pub_sub/listener.ex For now all that this does is listen on a channel provided and log the messages broadcast to that channel. Next lets configure this as a Worker and give it the channel it will listen to. Hook up the listener Update the file so that it starts the we just defined with the appropriate channel. application.ex Listener users_changes See it in Action You should now be able to start your application again by running: mix ecto.migratemix phx.server Then any change you make to the table via Postgres’s CLI should be output to our Applications . users stdout Listening for Specific Actions While it’s nice to be able to see that occurs in the database often we only care about a small subset of these changes. Specifically for this demo: everything User created User subscription updated Luckily Elixir’s pattern matching provides a wonderfully simple way to cut down on all the noise and focus exactly on the shape of data you’re looking for. What we want to do is update to send the payload to a function that will match on the events that matter. handle_info The above only acts on only the exact events we are looking for and outputs a specific message to our log. Anything else is ignored. Link it up to an email service Standard out is nice but in a running application you don’t want to be combing logs for events that your business cares about, you want to be notified in realtime. Lets hook our messages up to an email provider so that we get a notification right in our inbox when a user signs up or upgrades their plan. The Elixir library and service make this far too easy. Bamboo Send Grid Setup Bamboo Add the following to your file config.exs Note: Bamboo offers many more adapters than just Send Grid if you have a different preference. https://github.com/thoughtbot/bamboo#adapters Create a file in your folder mailer.ex pub_sub_demo Sending emails from our Listener Update in to send our log messages to our own email address. handle_user_changes listener.ex Run it for a Final Time Now if you create or update a user in your table you should see emails appear in your SendGrid logs. users Thats it! Conclusion The concept of broadcasting events directly from your database isn’t something new and as a result this type of functionality is available to you in any language and framework. But! The core features and functionality of Elixir and Phoenix make using database events much easier and more reasonable than I have found in any other system. Pattern Matching and OTP principles are both simple and powerful and I encourage anyone to take a deep dive into them when possible. 🧞‍ This is open source! you can find it here on Github ❤️ I only write about programming and remote work. If you I won’t waste your time. follow me on Twitter

Get notified of user signups and plan changes automatically using Postgres & Phoenix PubSub

Photo by on . Olu Eletu Unsplash I feel extremely lucky to have been instructed in the basics of Git-etiquette from essentially the first day in my journey to become a software engineer (Thank you, !). A seemingly minor concept, was one of the first things taught to me as a budding developer, and accordingly I placed upon it the importance that was conveyed to me in that education. Krailo Surprisingly, it seems to me that there are many developers who appear to be unfamiliar with the of possibly the most ubiquitous tool in software development, Git. Since many programmers learn Git on-the-job, they are likely not coached in best practices for something considered more of a soft-skill, and it’s doubtful that anyone chooses to use poor git-etiquette. Nonetheless, the problem is still rampant amongst developers of all skill-levels. dos and don’ts Don’t worry though, the solution will not require hours of reading and completely changing your workflow. In fact, it’s a fairly easy problem to solve, and it just requires that you follow some easy to remember guidelines. So let’s get started with a few of the basics. Knowledge and implementation of these concepts will help you make the first steps from greeny-junior to Git-guru. Use the CLI and an editor… ..and stop using git commit -m “Useless message here" Trying to fit a well-formed commit message into a single command doesn’t work. A commit message is supposed to take a minute. You should have to stop and reason about the work you just did in order to explain it to others who may work on the project as well as you future self should you need to revisit these changes. Writing a well-thought-out message is your way of writing into “history” the changes you have made and explaining what you changed, and why. When you take the time to compose your messages in a text-editor window(as opposed to on the command line) it will cause you to slow down and make sure your changes are complete and worth sharing, as well as allow you to be more descriptive in your explanation of why the changes were necessary. When you use the editor to compose messages, it can help catch errors in your code much sooner. For example, I use the git aliases in my work flow. The command expands into which will open your default editor with a summary of your current changes in the form of comments. What’s staged, what isn’t, untracked files, and a diff of the changes you are currently committing, allowing you to review the changes as you are writing your commit message. Oh-My-Zsh gc git commit --verbose Reading through the diffs before you write your message can save you from committing ‘debugger’ code and/or help spot a typo/spelling error that normally wouldn’t get caught until code review. : Tip Follow proper commit message format The Title-line When composing a title-line, I try and follow : 3 rules The title-line starts with a capital letter, is 50 characters(or less), and has no punctuation at the end. It should be in the imperative voice. i.e. ‘Add new About Us page’ or ‘Refactor tests for the order model’ It should correctly complete the sentence, “If accepted, this commit will <your commit message goes here>.” A super-basic commit message using proper grammar and structure. (`git commit — verbose` usingNeovim) The Body A brief summary of the changes. If the changes were simple enough they can be understood with just a “Title” line then you may opt out of this section(the above picture is a good example of changes that don’t need a detailed explanation). The body should explain the “what” and the “why” for your changes, not the “how.” Anyone who reads the code should be able to discern how you implemented your change. Use the body section to cover the parts that might not be so obvious. An example of pretty thorough commit messages with explanatory body sections If you are using a ticket system(Trello, Jira, etc.), . It’s a nice reference to have for anyone reviewing your PRs who may be unfamiliar with the issue or for other developers working on the same project. Tip: management INCLUDE THE LINK TO THE CARD/ISSUE Be atomic and keep changes together in related commits Group like changes together in the same commit Not even close to logical grouping Don’t throw some config changes and new CSS in the same commit. Break things apart in a logical manner.(Remember, even if you do more than one chunk of changes at a time, you can still commit them in a sensible manner, grouped by like-changes) The same changes from the previous image, but broken out into more “atomic” commits. Your commits show the progression of your code. Make it a story that is easy to follow. Even if it doesn’t seem important right now, being able to follow the history of your code can help you locate the source of regressions through logical deduction( ). that JS error in the console likely isn’t from the commit where you updated CSS and changed some icons in the footer The Git log from a feature I recently worked on. This by no means is all you need to know to use git masterfully, but it is a solid foundation from which to build. Knowledge of your and how best to use them is just one facet of becoming a high-quality developer. Attention to these “small” things will help you become more productive, and make working with you much more enjoyable. Even if the only person reaping those benefits is ten-weeks-from-now-You. tools Yes, something as small as the form of your commit messages may seem unimportant or tedious, but attention to the small things and fostering good habits is key to success. “We are what we repeatedly do; Excellence, then, is not an act but a habit.” — Aristotle For further reading, check out: This which, along with , are the sources for most of the material in this article. blog post by Tim Pope this blog post by Chris Beams for is a definite read. Caleb Thompson’s article Thoughtbot must The book . === “Pro Git” by Chacon & Straub GitBible s by Atlassian is frequently my go-to reference for some of the more advanced git-topics(**really solid examples and explanations**). This set of tutorial If you liked this article, please hit that 👏 button and consider following me. The more people that show interest in what I write about, the more likely I am to keep producing. Thanks!

Stories, News & Mentions about THOUGHTSPOT

Thoughtspot’s Stories on HackerNoon