FansForX, provides ready-to-deploy onlyfans clone script with cutting-edge features like- in-app chat, video recommendation, payment gateway, etc. Besides, content creators can roll out various subscription bundles based on their preferences. It is a highly customizable interactive feed for ...

FansForX - Crunchbase Company Profile & Funding

OnlyFans clone is a creator platform where user pays to view content. FansForX gives 100% source code with 1 Year Free support for OnlyFans script at 499$

OnlyFans Clone - FansForX - OnlyFans Clone Script

&quot;FansForX is one of the leading Onlyfans clone app development companies. They are an expert team of techies who are highly proficient in building clone apps. They work round the clock to provide robust and state-of-the-art clone apps across various time zones.

FansForX Company Profile, information, investors, valuation & Funding

Is fansforx.com legit or a scam? Read reviews, company details, technical analysis, and more to help you decide if this site is trustworthy or fraudulent.

fansforx.com Reviews: Is this site a scam or legit? - Scam Detector

Stay updated with our blogs to know the latest trends and news related to social media , onlyfans, and content subscriptions related topics.

Blogs - FansForX | OnlyFans Clone

A chic OnlyFans Clone Script that is quicker and convenient than ever. FansForX is one of the leading Onlyfans clone app development companies. We are an expert team of techies who are highly proficient in building clone apps.

FansForX

FansForX is one of the leading Onlyfans clone app development companies. We are an expert team of techies who are highly proficient in building clone apps. We work round the clock to provide robust and state-of-the-art clone apps across various time zones. We have built successful clone scripts ...

FansForX | StartUs

Twitter

FansForX | Bark Profile

The review of fansforx.com has been based on an analysis of 40 facts found online in public sources. Sources we use are if the website is listed on phishing and spam sites, if it serves malware, the country the company is based, the reviews found on other sites, and many other facts.

fansforx.com Reviews | check if site is scam or legit| Scamadviser

FansForX, provides ready-to-deploy onlyfans clone script with cutting-edge features.

FansForX - Tech Stack, Apps, Patents & Trademarks

Learn about FansForX. Read FansForX reviews from real users, and view pricing and features of the Creator Economy software

FansForX Reviews and Pricing 2023

Log in or sign up to view

Do you agree with fansforu.com&#x27;s TrustScore? Voice your opinion today and hear what 4 customers have already said.

fansforu.com Reviews | Read Customer Service Reviews of fansforu.com

Discover stories by FansForX on Vocal. OnlyFans clone app development company: https://fansforx.com

FansForX | Vocal

Ratings &amp; reviews for FansForX - OnlyFans Clone (San Diego), FansForX is one of the leading Onlyfans clone app development companies.

Review profile of FansForX - OnlyFans Clone | ProvenExpert.com

FansForX is a white-labeled and state-of-the-art Onlyfans clone app. Our professionals are the best in the web and mobile app development business. Our Onlyfans clone app is nothing short of excellent as we use the best of the best tech stack. We at FansForX, work together with one goal, which is to provide the best products and customer service.

When talking about data processing, people often abbreviate it as “ETL.” However, if we look closely, data processing has undergone several iterations from ETL, ELT, XX ETL (such as Reverse ETL, Zero-ETL) to the currently popular EtLT architecture. While the Hadoop era mainly relied on ELT (Extract, Load, Transform) methods, the rise of real-time data warehouses and data lakes has rendered ELT obsolete. EtLT has emerged as the standard architecture for real-time data loading into data lakes and real-time data warehouses. Let’s explore the reasons behind the emergence of these architectures, their strengths and weaknesses, and why EtLT is gradually replacing ETL and ELT as the global mainstream data processing architecture, along with practical open-source methods. ETL Era (1990–2015) In the early days of data warehousing, Bill Inmmon, the proponent of data warehousing, defined it as a data storage architecture for partitioned subjects, where data was categorized and cleaned during storage. During this period, most data sources were structured databases (e.g., MySQL, Oracle, SQLServer, ERP, CRM), and data warehouses predominantly relied on OLTP databases (e.g., DB2, Oracle) for querying and historical storage. Handling complex ETL processes with such databases proved to be challenging. To address this, a plethora of ETL software emerged, such as Informatica, Talend, and Kettle, which greatly facilitated integrating complex data sources and offloading data warehouse workloads. Advantages: Clear technical architecture, smooth integration of complex data sources, and about 50% of the data warehouse work handled by ETL software. Disadvantages: All processing is done by data engineers, leading to longer fulfillment of business requirements; high hardware costs with double investments, especially when dealing with large data volumes. During the early and mid stages of data warehousing, when data source complexity was significantly higher, the ETL architecture became the industry standard and remained popular for over two decades. ELT Era (2005–2020) As data volumes grew, both data warehouse and ETL hardware costs escalated. New MPP (Massively Parallel Processing) and distributed technologies emerged, leading to the gradual shift from ETL to ELT architecture in the later stages of data warehousing and the rise of big data. Teradata, one of the major data warehouse vendors, and Hadoop Hive, a popular Hadoop-based data warehousing solution, adopted ELT architecture. They focused on direct loading (without complex transformation like join and group) of data into the data warehouse’s data staging layer, followed by further processing using SQL or H-SQL from the staging layer to the data atomic layer and finally to the summary layer and indicator layer. While Teradata targeted structured data and Hadoop targeted unstructured data, they adopted similar 3–4 layer data storage architectures and methodologies for data warehousing globally. Advantages: Utilizing data warehouse high-performance computing for large data volume processing, resulting in higher hardware ROI. Complex business logic can be handled by SQL, reducing the overall cost of data processing personnel by employing data analysts and business-savvy technical personnel, without the need to understand ETL tools like Spark and MapReduce. Disadvantages: Suitable for simple and large data volume scenarios. Inadequate for complex data sources and lacking support for real-time data warehousing requirements. ODS (Operational Data Store) was introduced as a transitional solution to handle complex data sources that ELT-based data warehouses couldn’t load and to improve real-time capabilities. ODS involved processing complex data sources through real-time CDC (Change Data Capture), real-time APIs, or short-batch processing (Micro-Batch) into a separate storage layer before ELT-ing them into the enterprise data warehouse. Currently, many enterprises still adopt this approach. Some companies place the Operational Data Store (ODS) within the data warehouse and use Spark and MapReduce for initial ETL (Extract, Transform, Load) processes. Later, they perform business data processing within the data warehouse (using tools like Hive, Teradata, Oracle, and DB2). At this stage, the early EtLT (Extract, Transform, Load, and Transform) community has already formed. It is characterized by a division of roles, where the complex processes of data extraction, Change Data Capture (CDC), data structuring, and standardization are often handled by data engineers, referred to as “t.” Their objective is to move data from the source system to the underlying data preparation layer or data atomic layer within the data warehouse. On the other hand, the processing of complex data atomic layers with business attributes, data aggregation, and generating data metrics (involving operations such as Group by and Join) is typically performed by business data engineers or data analysts who are skilled in using SQL. As a result of the emergence of the EtLT architecture, standalone projects like ODS have gradually faded out of the limelight due to the increase in data volume and the adoption of EtLT principles. EtLT Era (2020-Present) The EtLT architecture, as summarized by James Densmore in the , is a modern and globally popular data processing framework. EtLT emerged in response to the transformations in the modern data infrastructure. Data Pipelines Pocket Reference 2021 Background of the EtLT Architecture The modern data infrastructure has the following characteristics, which led to the emergence of the EtLT architecture: Cloud, SaaS, and Hybrid Local Data Sources Data Lake and Real-time Data Warehouses New Generation Big Data Federation Proliferation of AI Applications Fragmentation of Enterprise Data Community The appearance of complex data sources In the current global enterprise landscape, the advent of cloud and SaaS has made an already complex data source environment even more intricate. Dealing with SaaS data has given rise to a new concept of data ingestion, exemplified by tools like Fivetran and Airbyte, aiming to address the ELT (Extract, Load, Transform) challenges of ingesting SaaS data into data warehouses like Snowflake. Additionally, the complexity of data sources has increased with the proliferation of cloud-based data storage services (e.g., AWS Aurora, AWS RDS, MongoDB Service) and the coexistence of traditional on-premises databases and software (SAP, Oracle, DB2) in hybrid cloud architectures. Traditional ETL and ELT architectures are unable to cope with the intricacies of processing data in such a complex environment. Data Lake and Real-time Data Warehouses In modern data architecture, the emergence of data lakes has combined the features of traditional ODS (Operational Data Store) and data warehouses. Data lakes enable real-time data processing and facilitate data changes at the source (e.g., Apache Hudi, Apache Iceberg, Databricks Delta Lake). Simultaneously, the concept of real-time data warehouses has surfaced, with various new computing engines (e.g., Apache Pinot, ClickHouse, Apache Doris) making real-time ETL a priority. However, traditional CDC ETL tools or real-time stream processing platforms face challenges in providing adequate support for data lakes and real-time data warehouses, either due to compatibility issues with new storage engines or limitations in connecting to modern data sources, lacking robust architecture and tool support. The Emergence of a New Generation of Big Data Federation In modern data architecture, a new breed of architectures has emerged, aiming to minimize data movement across different data stores and enabling complex queries directly through connectors or rapid data loading. Examples include Starburst’s TrinoDB (formerly PrestoDB) and OneHouse based on Apache Hudi. These tools excel at data caching and on-the-fly cross-data-source queries, making them unsuitable for support by traditional ETL/ELT tools in this new Big Data Federation paradigm. The Rise of Large-scale Models With the emergence of ChatGPT in 2022, AI models have become algorithmically feasible for widespread enterprise applications. The bottleneck for AI application deployment now lies in data supply, which has been addressed by data lakes and Big Data Federation for data storage and querying. However, traditional ETL, ELT, and stream processing have become bottlenecks for data supply, either unable to quickly integrate various complex traditional and emerging data sources or failing to support diverse data requirements for both AI training and online AI applications using a single codebase. Fragmentation of the Enterprise Data Community As data-driven approaches become more ingrained in enterprises, the number of data users within organizations has rapidly increased. These users range from traditional data engineers to data analysts, AI practitioners, sales analysts, and financial analysts, each with diverse data requirements. After experiencing various shifts like No-SQL and New-SQL, SQL has emerged as the sole standard for complex business analysis. A considerable number of analysts and business unit engineers now use SQL to address the “last mile” problem of data analysis within enterprises. Meanwhile, the handling of complex unstructured data is left to professional data engineers using technologies like Spark, MapReduce, and Flink. Consequently, the demands of these two groups diverge significantly, making traditional ETL and ELT architectures inadequate to meet the needs of modern enterprise users. EtLT Architecture Emerges! Against the backdrop mentioned above, data processing has gradually evolved into the EtLT architecture: EtLT Architecture Overview: EtLT splits the traditional ETL and ELT structures and combines real-time and batch processing to accommodate real-time data warehouses and AI application requirements. E(xtract) — Extraction: EtLT supports traditional on-premises databases, files, and software as well as emerging cloud databases, SaaS software APIs, and serverless data sources. It can perform real-time CDC on database binlog and real-time stream processing (e.g., Kafka Streaming) and also handle bulk data reading (multi-threaded partition reading, rate limiting, etc.). t(ransform) — Normalization: In addition to ETL and ELT, EtLT introduces a small “t,” focusing on data normalization. It rapidly converts complex and heterogeneous extracted data sources into structured data that can be readily loaded into the target data storage. It deals with real-time CDC by splitting, filtering, and changing field formats, supporting both batch and real-time distribution to the final Load stage. L(oad) — Loading: The loading stage is no longer just about data loading but also involves adapting data source structures and content to suit the target data destination (Data Target). It should handle data structure changes (Schema Evolution) in the source and support efficient loading methods such as Bulk Load, SaaS loading (Reverse ETL), and JDBC loading. EtLT ensures support for real-time data and data structure changes, along with fast batch data loading. (T)ransform — Conversion: In cloud data warehouses, on-premises data warehouses, or new data federations, business logic is processed. This is typically achieved using SQL, either in real-time or batch mode, to transform complex business rules accurately and quickly into data usable by business or AI applications. In the EtLT architecture, different user roles have distinct responsibilities: EtL Phase: Primarily handled by data engineers who convert complex and heterogeneous data sources into data that can be loaded into the data warehouse or data federation. They do not need to have an in-depth understanding of enterprise metric calculation rules but must be proficient in transforming various source and unstructured data into structured data. Their focus is on data timeliness and the accuracy of transforming source data into structured data. T Phase: Led by data analysts, business SQL developers, and AI engineers who possess a deep understanding of enterprise business rules. They convert business rules into SQL statements to perform analysis and statistics on the underlying structured data, ultimately achieving data analysis within the enterprise and enabling AI applications. Their focus is on data logic relationships, data quality, and meeting business requirements for final data results. Open-Source Implementation of EtLT There are several open-source implementations of EtLT in modern data architecture. Examples include DBT, which helps analysts and business developers quickly develop data applications based on Snowflake, and Apache DolphinScheduler, a visual workflow orchestration tool for big data tasks. DolphinScheduler plans to introduce a Task IDE, allowing data analysts to directly debug SQL tasks for Hudi, Hive, Presto, ClickHouse, and more, and create workflow tasks through drag-and-drop. As a representative of the EtLT architecture, Apache SeaTunnel started with support for various cloud and on-premises data sources, gradually expanding its capabilities to include SaaS and Reverse ETL, as well as accommodating the demands of large-scale model data supply. It has been continually refining the EtLT landscape. The latest SeaTunnel Zeta computing engine delegates complex operations such as Join and Groupby to the ultimate data warehouse endpoint, focusing on data normalization and standardization. This approach aims to achieve the goal of unified real-time and batch data processing with a single set of code and a high-performance engine. Additionally, SeaTunnel now includes support for large-scale models, making it possible for these models to directly interact with over 100 supported data sources (refer to ), ranging from traditional databases to cloud databases and ultimately SaaS. https://seatunnel.apache.org/docs/Connector-v2-release-state/ Since joining the Apache Incubator in late 2022, Apache SeaTunnel has witnessed a five-fold growth in one year, and currently, it supports more than 100 data sources (as seen at ). The connector support has been progressively improved, encompassing traditional databases, cloud databases, and SaaS offerings. https://seatunnel.apache.org/docs/2.3.2/category/source-v2 The release of SeaTunnel Zeta Engine in Apache SeaTunnel 2.3.0 brings features such as data distributed CDC, schema evolution for target source data tables, and the synchronization of entire databases and multiple tables. Its excellent performance has garnered attention from numerous global users, including Bharti Airtel, the second-largest telecommunications operator in India, Shopee.com, an e-commerce platform in Singapore, and Vip.com, a major online retailer. Large Scale Model Training support One noteworthy aspect is that SeaTunnel now offers support for large-scale model training and vector databases, enabling seamless interactions between large models and the 100+ supported data sources of SeaTunnel (see the article ). Furthermore, SeaTunnel can leverage ChatGPT to directly generate SaaS Connectors, facilitating rapid access to a wide range of internet information for your large-scale models and data warehouses. Breakthrough in the book search field! Use Apache SeaTunnel, Milvus, and OpenAI to improve the accuracy and efficiency of book title similarity search As the complexity of AI, cloud, and SaaS continues to increase, the demand for real-time CDC, SaaS, data lakes, and real-time data warehouse loading has made simple ETL architectures inadequate to meet the needs of modern enterprises. EtLT architecture, tailored for different stages of enterprise development, is destined to shine in the modern data infrastructure. With the mission of “Connecting the world’s data sources and synchronizing them as swiftly as flying”, Apache SeaTunnel warrants the attention of all data professionals. Also published . here

ELT is Dead, and EtLT Will End Modern Data Processing Architecture

Community Violence Intervention Programs: Are They Effective?

ChatGPT was released in June 2020 and it is developed by OpenAI. It has led to revolutionary developments in many areas. One of these areas is the creation of database queries.The database queries can be generated from plain text via . It helps you with databases that you have little knowledge about. ChatGPT Artificial intelligence, which has progressively integrated into our daily lives, has gained significant momentum with the rise of ChatGPT. Before moving into the project, let’s briefly explore ChatGPT. To fully comprehend the capabilities of ChatGPT, it is essential to have a solid grasp of its underlying architecture. ChatGPT has been developed based on the architecture. So let’s take a look at transformers first. GPT For direct access to the project, click . here Transformers utilize seq2seq frameworks, enabling the conversion of one sequence into another. The sequences have defined ordering. As an illustration, sentences can be viewed as a succession of words. The Transformers can also be used to solve NLP such as text translation. We need a lot of labeled data for training this architecture. This would be difficult for transformers to learn. We can use transfer learning to solve this problem. Transformers consist of two components (Fig 1): an encoder and a decoder, both adept at acquiring a proficient language representation. This proficiency allows us to construct language models from each component. By stacking the encoders, we obtain a idirectional ncoder epresentation of ransformers, commonly known as . Similarly, by stacking the decoder units, we can achieve enerative re- raining that is GPT. In the our case we are focusing on GPT. Let’s think about transfer learning in the context of GPT. When we train a model from scratch, it typically demands a substantial amount of data as the parameters are initially set randomly. However, imagine a scenario where the parameters fortuitously align with the values we require. In such cases, we don’t necessitate an extensive dataset to attain our desired outcome. As will be understood, BERT and GPT are used in transfer learning concept. B E R T BERT G P T The GPT training is thus divided into two parts. We have pre-training part where we train the GPT architecture to understand what language is and then fine tuning part where we use transfer learning to further train the GPT architecture to perform well on specific language tasks GPT has a structure that takes random word sequences as input and predicts the next most suitable word. Sample prediction can be seen at Fig 2. Language modeling is selected as an ideal foundation for understanding the fundamental aspects of language and can be readily fine-tuned. It is commonly known as a self-supervised task since the sentences themselves serve as both input and output labels. Let’s move on to ChatGPT. As it can be seen in Fig 3., the entire ChatGPT process can be divided into three major steps. In the first step, GPT model is used that has been pre-trained on understanding language itself. The next step involves fine-tuning the model to effectively process user prompts and generate appropriate responses based on those prompts. To facilitate this process, labeled data is obtained by engaging labelers who not only provide prompts but also specify the desired response for each prompt. This allows for engaging in supervised fine-tuning of the GPT model, as there are both input prompts and corresponding output responses. In the next step, the supervised fine-tuned model is used that it was obtained from first step. A single prompt is passed through the model, and multiple responses are generated. The labeler then evaluates and ranks the quality of these responses. Each generated response is assigned a corresponding reward, which is utilized to train another GPT model known as the rewards model. Being a model itself, the rewards model functions by taking an initial prompt and one of the responses as input and generating a reward as output. This reward quantifies the quality or goodness of the response. In the third step, an unseen prompt will be taken and passed through a replica of the supervised fine-tuned model. This will generate a response, which will then be evaluated using a rewards model to determine its rank or quality. The rank obtained will be used to further refine our already fine-tuned model. This is achieved by incorporating the rank into the loss function of the PPO model, allowing updates to the model’s parameters through backpropagation. What’s particularly fascinating is that this process not only helps the model exhibit non-toxic behavior but also enables it to generate factual responses. This is because the reward itself was generated based on the quality of the response. ChatSQL Now let’s make a project with ChatGPT using python. LangChain framework is used for this project. Our objective is to retrieve information from our database using plain text. In our work, we utilized MySQL as the target database. Nevertheless, this approach can also be used to generate queries on other databases. The sample database is shown below. ( All codes can be found here https://github.com/ademakdogan/ChatSQL ) +-----+--------------------------------------------------------+------------------------+-------------------+--------+------------------+
| ID | Title | Author | Genre | Height | Publisher |
+-----+--------------------------------------------------------+------------------------+-------------------+--------+------------------+
| 1 | Fundamentals of Wavelets | Goswami, Jaideva | signal_processing | 228 | Wiley |
| 2 | Data Smart | Foreman, John | data_science | 235 | Wiley |
| 3 | God Created the Integers | Hawking, Stephen | mathematics | 197 | Penguin |
| 4 | Superfreakonomics | Dubner, Stephen | economics | 179 | HarperCollins |
| 5 | Orientalism | Said, Edward | history | 197 | Penguin |
| 6 | Nature of Statistical Learning Theory, The | Vapnik, Vladimir | data_science | 230 | Springer |
| 7 | Integration of the Indian States | Menon, V P | history | 217 | Orient Blackswan |
| 8 | Drunkard's Walk, The | Mlodinow, Leonard | science | 197 | Penguin |
| 9 | Image Processing & Mathematical Morphology | Shih, Frank | signal_processing | 241 | CRC |
| 10 | How to Think Like Sherlock Holmes | Konnikova, Maria | psychology | 240 | Penguin |
| 11 | Data Scientists at Work | Sebastian Gutierrez | data_science | 230 | Apress |
| 12 | Slaughterhouse Five | Vonnegut, Kurt | fiction | 198 | Random House |
| 13 | Birth of a Theorem | Villani, Cedric | mathematics | 234 | Bodley Head |
| 14 | Structure & Interpretation of Computer Programs | Sussman, Gerald | computer_science | 240 | MIT Press |
| 15 | Age of Wrath, The | Eraly, Abraham | history | 238 | Penguin |
| 16 | Trial, The | Kafka, Frank | fiction | 198 | Random House |
| 17 | Statistical Decision Theory' | Pratt, John | data_science | 236 | MIT Press |
| 18 | Data Mining Handbook | Nisbet, Robert | data_science | 242 | Apress |
| 19 | New Machiavelli, The | Wells, H. G. | fiction | 180 | Penguin |
| 20 | Physics & Philosophy | Heisenberg, Werner | science | 197 | Penguin |
| 21 | Making Software | Oram, Andy | computer_science | 232 | O'Reilly |
| . | ....... | ....... | .... | ... | .... |
| . | ....... | ....... | .... | ... | .... | The ChatSQL consists of two main parts. First of all, mysql queries are generated from the given plain text via ChatGPT. These generated queries are executed. In the second step, the results returned in the database are evaluated. At this stage, the results can either be presented directly to the user or sent back to ChatGPT for further analysis and interpretation. So you can chat about your database with ChatGPT. Suppose the user wants “Show me the book type fiction which they height bigger than 175 and smaller than 178. The author shouldn’t be ‘Doyle, Arthur Conan’.”. So the following command can be used in this case. python3 chatsql.py -p 'Show me the book type fiction which they height bigger than 175 and smaller than 178. The author shouldn't be 'Doyle, Arthur Conan'. ' As a result, the following information is obtained. CHATGPT QUERY------------------:
SELECT * FROM bt WHERE Genre = 'Fiction' AND Height > 175 AND Height < 178 AND Author != 'Doyle, Arthur Conan'
RAW RESULT------------------:
[(32, 'Pillars of the Earth, The', 'Follett, Ken', 'fiction', 176, 'Random House'), (37, 'Veteran, The', 'Forsyth, Frederick', 'fiction', 177, 'Transworld'), (38, 'False Impressions', 'Archer, Jeffery', 'fiction', 177, 'Pan'), (72, 'Prisoner of Birth, A', 'Archer, Jeffery', 'fiction', 176, 'Pan'), (87, 'City of Joy, The', 'Lapierre, Dominique', 'fiction', 177, 'vikas'), (128, 'Rosy is My Relative', 'Durrell, Gerald', 'fiction', 176, 'nan')]
PROCESSED RESULT------------------ :
The books 'Pillars of the Earth, The' by Ken Follett, 'Veteran, The' by Frederick Forsyth, 'False Impressions' by Jeffery Archer, 'Prisoner of Birth, A' by Jeffery Archer, 'City of Joy, The' by Dominique Lapierre, and 'Rosy is My Relative' by Gerald Durrell are all fiction books with 176 or 177 pages published by Random House, Transworld, Pan, Vikas, and Nan, respectively. If your database columns are named appropriately, chatgpt will understand these names and respond queries accordingly. However, there may be instances where the column names in the database are not meaningful or ChatGPT may not fully figure out their context. Hence, in order to ensure the system functions correctly, It is necessary to provide ChatGPT with prior information about the database. Info.json file can be used to adding information. +-----+--------------------------------------------------------+------------------------+-------------------+------+------------------+
| ID | aa | bb | cc | dd | ee |
+-----+--------------------------------------------------------+------------------------+-------------------+------+------------------+
| 1 | Fundamentals of Wavelets | Goswami, Jaideva | signal_processing | 228 | Wiley |
| 2 | Data Smart | Foreman, John | data_science | 235 | Wiley |
| 3 | God Created the Integers | Hawking, Stephen | mathematics | 197 | Penguin |
| 4 | Superfreakonomics | Dubner, Stephen | economics | 179 | HarperCollins |
| 5 | Orientalism | Said, Edward | history | 197 | Penguin |
| . | ....... | ....... | .... | ... | .... |
| . | ....... | ....... | .... | ... | .... | As an example, let’s have a badly named database as below. In this case, the required information about the database is inputted into the info.json file. {“bt”: “Table Name”, “aa”: “Title of the book”, “bb”: “Author of the book”, “cc”: “Type of book”, “dd”: “Height of the book”, “ee”: “Book Publisher”} and then use the same command: python3 chatsql.py -p 'Show me the book type fiction which they height bigger than 175 and smaller than 178. The author shouldn't be 'Doyle, Arthur Conan'. ' Even if the column names in the database are poorly selected, chatgpt generates the correct query because we provide the correct information {'query': "SELECT aa, bb, cc, dd FROM bt WHERE cc = 'fiction' AND dd > 175 AND dd < 178 AND bb != 'Doyle, Arthur Conan'", 'raw_result': "[('Pillars of the Earth, The', 'Follett, Ken', 'fiction', 176), ('Veteran, The', 'Forsyth, Frederick', 'fiction', 177), ('False Impressions', 'Archer, Jeffery', 'fiction', 177), ('Prisoner of Birth, A', 'Archer, Jeffery', 'fiction', 176), ('City of Joy, The', 'Lapierre, Dominique', 'fiction', 177), ('Rosy is My Relative', 'Durrell, Gerald', 'fiction', 176)]", 'processed_result': '\nThe books "Pillars of the Earth, The" by Ken Follett, "Veteran, The" by Frederick Forsyth, "False Impressions" by Jeffery Archer, "Prisoner of Birth, A" by Jeffery Archer, "City of Joy, The" by Dominique Lapierre and "Rosy is My Relative" by Gerald Durrell are all fiction and have page lengths of 176 or 177.'} The next project could be on generating queries (mongo, sql) from prompts with free models (Llama). Project Repo : https://github.com/ademakdogan/ChatSQL Github : https://github.com/ademakdogan Linkedin : https://www.linkedin.com/in/adem-akdoğan-948334177/ References [1] Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P. and Amodei, D. (2020). Language models are few-shot learners. arXiv preprint arXiv:2005.14165. [2] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI Blog. [3] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N. and Polosukhin, I. (2017). Attention is all you need. In Advances in neural information processing systems (pp. 5998–6008). [4] Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Vol. 1, pp. 4171–4186). [5] Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., & Zettlemoyer, L. (2019). BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461. [6] Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., … & Liu, P. J. (2019). Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. [7] Vaswani, A., & et al. (2017). Transformer-XL: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860.

ChatSQL: Enabling ChatGPT to Generate SQL Queries from Plain Text

চ্যাটএসকিউএল: প্লেইন টেক্সট থেকে এসকিউএল কোয়েরি তৈরি করতে ChatGPT সক্ষম করা

ChatSQL: ChatGPT aktivieren, um SQL-Abfragen aus reinem Text zu generieren

ChatSQL: habilitar ChatGPT para generar consultas SQL a partir de texto sin formato

ChatSQL – Activation de ChatGPT pour générer des requêtes SQL à partir de texte brut

ChatSQL: प्लेन टेक्स्ट से SQL क्वेरी जनरेट करने के लिए ChatGPT को सक्षम करना

ChatSQL: ChatGPT を有効にしてプレーン テキストから SQL クエリを生成する

ChatSQL: ChatGPT를 활성화하여 일반 텍스트에서 SQL 쿼리 생성

ChatSQL : habilitando o ChatGPT para gerar consultas SQL a partir de texto simples

ChatSQL: ChatGPT'nin Düz Metinden SQL Sorguları Oluşturmasını Etkinleştirme

ChatSQL: Bật ChatGPT để tạo truy vấn SQL từ văn bản thuần túy

ChatSQL：启用 ChatGPT 从纯文本生成 SQL 查询

Tranformer-based vision models are gradually evolving and are reported to be as good as Convolutional models on Classification, Segmentations and Action Recognition tasks. We have a whole array of Convolutional models for vision tasks and are more popular than Transformer-based ones. This blog delves into the SWin Transformer Vision model that was presented in the International Conference for Computer Vision (ICCV) 2021 by the Microsoft Research Team. It benchmarks its performance against several SOTA Convolution-based models on the Dog Breed Image Classification task. Will transformer-based models become the next big thing in Computer Vision? With transformers being a successful solution for language tasks, will it unify the various AI subfields and present powerful solutions to more complex problems? So rolling up my sleeves to evaluate how good they are on the classification task to begin with. Chosen Data and its Problem Statement: The myriad of dog breeds with subtle changes in their physical appearances have been a challenge to veterinarians, dog owners, animal shelter staff and potential dog owners in identifying their right breed. They need to identify the breed in order to provide appropriate training, treatment and meet their nutritional needs.The data is sourced from the that contains ~20K images of 120 breeds of dogs across the world. This data has been split almost equally into train and test set for a Kaggle Competition . Stanford Dog Dataset Dog Breed Identification The objective of the solution is to build a dog breed identification system capable of correctly identifying dog breeds with minimal data and rightly identifying even similar looking dog breeds. This is  a multi-class classification task and for every image, the model has to predict probability for all the 120 breeds. The one with the highest probability is the most probable breed of the dog in the image. Exploratory Data Analysis For a breed, the training data has a maximum of 120+ images and a minimum of 66 images. Each class, on an average is expected to have 85 images. Looking at the bar plot, it is concluded that there are no class imbalances. Though there is no class imbalance, data may be insufficient to train the neural network. Image Augmentation using random image perturbations and pre-trained models will be able to circumvent this problem. Top 5 breeds with the most images are scottish_deerhound, maltese_dog, afghan_hound, entlebucher and bernese_mountain_dog. Bottom 5 breeds with the least images are golden_retriever, brabancon_griffon, komondor, eskimo_dog and briard. A quick analysis on spatial dimensions of the training images is done to understand the distribution of image height, width and their aspect ratios. Images with very low (<0.5) and very high (>2.3) aspect ratios are considered to be anomaly images. 1.5 is considered to be a good aspect ratio. In analysing various dog breeds, below pairs of breeds were generally found to look alike. 'boston_bull', 'french_bulldog' 'beagle', 'english_foxhound' 'bernese_mountain_dog', 'greater_swiss_mountain_dog' 'malamute', 'siberian_husky' 'basenji', 'ibizan_hound' 'vizsla', 'rhodesian_ridgeback' 'lhasa', 'shih-tzu' 'whippet', 'italian_greyhound' 'brittany_spaniel', 'welsh_springer_spaniel', 'blenheim_spaniel' 'malinois', 'german_shepherd' 'border_collie', 'collie' 'norfolk_terrier', 'norwich_terrier’ Evaluation Metrics LogLoss, which stringently evaluates the confidence of the model predictions for all 120 classes by comparing the prediction probability to the ground truth probability, is the key metric. N*N confusion matrix is used to study the outcome of the predictions to analyse if there are any pair of classes confused by the model and incorrectly predicted. SWIN - Transformer Based Model This architecture is based on developed by Microsoft Research Team. This paper discusses an improved ViT architecture that produces a hierarchical representation of feature maps reducing the computation complexity of self-attention mechanism from quadratic to linear. It is proven to give identical results as SOTA Convolution Networks like EfficientNet on Imagenet classification problem. “Swin Transformer: Hierarchical Vision Transformer using Shifted Windows” The building blocks of this architecture is explained in the below notes: Image Patching: In NLP, the tokens which are the processing elements of a model, are the words in a sentence where the size of 1 token is 1 (just 1 word). ViT ( sion ransformers) treat “image patches” as tokens where each patch is a partition of an image consisting of a group of neighbouring pixels. Each image patch is a token. The size of 1 token in any ViT is patch_height * patch_width * number of channels. Based on the patch dimensions, we get a number of patches or tokens for a single image. If the image size (H*W*3) is 224 * 224 * 3 pixels and the patch size is 4 * 4, then we get 224/4 * 224/4 = 56 * 56 patches or 3136 tokens from the image. Each token or patch will be of size 4*4*3 = 48 dimensions or 48 pixels of data. So the input to the architecture for this image consists of 3136 tokens each of size 48 dimensions. Vi T The underlying mechanism of the SWIN transformer is analogous to any CNN based architecture where the spatial dimensions of the image is decreased and the number of channels is increased. SWIN transformer at every stage in the hierarchical architecture, also reduces the number of image patches or the number of tokens while increasing the token dimensions. With this mechanism in mind, it is easier to understand the SWIN architecture easily. At every stage of the architecture, we can see the number of tokens decreasing while the token size is increasing. The SWIN-T architecture, apart from the “ is also made up of 3 other building blocks - Linear Embedding, Swin Transformer Block, Patch Merging. These building blocks are repeated and it process feature maps in a hierarchical fashion. Patch Partitioner”, Linear Embedding: The 3136 tokens each of 48 dimension from the “Patch Partitioner” are fed to a feed forward layer to embed the token of 48 feature vector into a feature vector of size ‘C’. ‘C’ here acts as the capacity of the transformer and the SWIN architecture has 4 variants based on it. Tiny (Swin-T)  - C is ‘96’ Small (Swin-S) - C is ‘96’ Base(Swin-B)  -  C is ‘128’ Large(Swin-L) - C is ‘192’ Image patching and linear embedding are jointly implemented in a single convolution whose kernel-size and stride-length is same as the patch-size. Number of channels in the convolutional will be ‘C’. SWin Transformer Block: The SWin Transformer Block is different from the standard transformer block in the ViT architecture. In SWin Transformers, the layer is replaced by either the or the . Multi-head Self Attention (MSA) Window MSA (W-MSA) module Shifted Window MSA (SW-MSA) module Stage1 consists of 2 SWIN-T Transformer Blocks (refer image) where the first Transformer Block has Window MSA (W-MSA) and the second Transformer Block has Shifted Window MSA (SW-MSA) module. In the SWin Transformer Block, the inputs and the outputs of the W-MSA and SW-MSA layers are passed via Normalization Layers. It is then subjected to a 2 layered Feed Forward Network with Gaussian Error Linear Units (GELU) activation. There are residual connections within each block and between these 2 blocks. Window MSA (W-MSA) and Shifted Window MSA (SW-MSA) modules Why is the standard attention layer in ViT replaced with the Windowed MSA layer? The standard attention layer in ViT was a global one calculating the attention of a patch with all other patches in the image thus leading to a quadratic complexity proportional to the image dimensions. This doesn't scale very well for high resolution images. The self-attention mechanism in the W-MSA or SW-MSA module is a local one that calculates self-attention only between patches within the same window of the image and not outside the windows. Windows are like larger partitions of the image where each window comprises of M*M patches. Replacing global self-attention with local self-attention reduced the computational complexity from quadratic to linear. Why both W-MSA and SW-MSA for local self-attention? What is the  difference between them? The key difference between W-MSA and SW-MSA attention modules is in the way how the windows for the image are configured. In W-MSA module, a regular window partitioning strategy is followed. The image is evenly partitioned into non-overlapping windows starting from the top-left pixel of the image, and each window contains M*M or M2 patches . In SW-MSA module, the window configuration is shifted from that of the W-MSA layer, by displacing the windows by (M/2, M/2) patches from the regular partitioning strategy. Why a shifted window partitioning strategy in SW-MSA? Since the attention is restricted locally within a window in W-MSA, the shifted window enables cross-window attention to still yield the benefits of global attention. This is possible because the boundaries of window1 in W-MSA layer are shared with windows W2, W4 and W5 in SW-MSA layer. Hence global attention happens indirectly via “local attention on shifted windows”. Patch Merging Layer Patch Merging layer reduces the number of tokens as the network gets deeper and deeper. The first patch merging layer concatenates the features of each group of 2×2 neighbouring patches. SWIN Transformers on Dog Breed Classification The package in PyPI has pretrained TF-Keras variants of the SWIN Transformers and is built based on the . Its code is available in . tfswin is used to train the dog breed images. tfswin official pytorch implementation github from tfswin import SwinTransformerBase224, preprocess_input
def build_model1(swintransformer):
 tf.keras.backend.clear_session()

 inputs = Input(shape=(resize_height, resize_width, 3))

 outputs = Lambda(preprocess_input)(inputs)
 outputs = swintransformer(outputs)
 outputs = Dense(num_classes, activation='softmax')(outputs)

 swin_model = Model(inputs=inputs, outputs=outputs)

 return swin_model

#build the model
swintransformer = SwinTransformerBase224(include_top=False,pooling='avg')
swin_model1 = build_model1(swintransformer)

#set the layers of the pretrained model as non-trainable
for layer in swin_model1.layers[2].layers: 
 layer.trainable = False

swin_model1.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),loss='categorical_crossentropy',metrics=['accuracy']) Convolutional Based Models To begin with, various ResNet variants were trained with different replacements for top layers, and partially to totally freezing the network. #Logloss of the test set using various ResNet variants
+------------+---------------+-------------------------+----------+
| Model Name | Retrained | Top Layers Replacement | Log_Loss |
+------------+---------------+-------------------------+----------+
| ResNet50 | None | ConvBlock_FC_Output | 0.96463 |
| ResNet50 | None | GlobalAvgPooling_Output | 0.58147 |
| ResNet50 | last 4 layers | ConvBlock_FC_Output | 2.10158 |
| ResNet50 | last 4 layers | GlobalAvgPooling_Output | 0.57019 |
+------------+---------------+-------------------------+----------+ Code corresponding to the ResNet50 model with least log loss from tensorflow.keras.layers import Input,Conv2D,Dense,BatchNormalization,Flatten,Concatenate, Dropout,MaxPooling2D
from tensorflow.keras.models import Model
from tensorflow.keras.applications.resnet50 import ResNet50, preprocess_input

def build_model():

 tf.keras.backend.clear_session()

 inputs = Input(shape=(resize_height, resize_width, 3))

 #added preprocess_input method as a layer to convert input images to those expected by Resnet
 processed_inputs = preprocess_input(inputs) 

 #use the pretrained ResNet model (Parameter pooling = 'avg' will take care of the Gobal Average Pooling of the ResNet model features)
 base_model = ResNet50(weights="imagenet", include_top=False,pooling='avg')(processed_inputs) 
 
 #output layer
 output = Dense(units=num_classes,activation='softmax',name='Output')(base_model)

 resnet_model = Model(inputs=inputs, outputs=output)
 return resnet_model

#build the model
resnet_model = build_model()

#set the layers of the resnet pretrained model as non-trainable except for its last 4 layers which needs to be re-trained for this data
for layer in resnet_model.layers[3].layers[:-4]: 
 layer.trainable = False

#compile the model
resnet_model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.0001),loss='categorical_crossentropy',metrics=['accuracy'])
print(resnet_model.summary())

history = resnet_model.fit(train_ds,
 epochs=50,
 validation_data=val_ds, callbacks=callbacks_list) More sophisticated pre-trained Convolutional models like EfficientNet, NASNet, InceptionResNet, Xception and InceptionV3 were also trained using the dog breed images both in standalone and also in an ensembled fashion. The log loss on the test data (consisting of 10K images) from these models were lower than any of the ResNet variants. #Logloss of the standalone model variants
+----------------------------+-------------+
| Model Name | Log_Loss |
+----------------------------+-------------+
| EfficientNetV2M | 0.28347 |
| Inception ResNet | 0.28623 |
| NasNetLarge | 0.33285 |
| Xception | 0.34187 |
| Inception_V3 | 0.54297 |
| EfficientNetV2M_GlobalAveg | 0.50423 |
| InceptionV3_GlobalAveg | 0.46402 |
+----------------------------+-------------+ For standalone models, the layers of the pre-trained models were frozen and their top layers were replaced by either Convolutional Layer followed by MaxPooling Layer Global Average Pooling Layer EfficientNet architecture is a rightly scaled model in all 3 dimensions - depth, width and resolution. NASNet-Large is the outcome of model architecture search done using various hyperparameter search algorithms, optimization algorithms and reinforcement learning algorithms. InceptionV3 factorized larger convolutions into smaller asynchronous convolutions and also included auxiliary classifiers to improve model convergence. Xception architecture does depth-wise separable convolutions instead of the inception modules in the Inception architecture. The log loss from the ensemble models reduced by > 0.1 than the best performing standalone model. +--------------------------------------------------------------------------+-----------+
| Model Name | Log_Loss |
+--------------------------------------------------------------------------+-----------+ 
| Ensemble1 - EfficientNEt,InceptionResNet,NasNet,Xception) | 0.17363 |
| Ensemble2 - EfficientNEt,InceptionResNet,NasNet,Xception and InceptionV3 | 0.16914 |
| Ensemble3 - Ensemble2 with 50% dropout. | 0.16678 |
| Ensemble4 - Ensemble of various EfficientNet Architecture | 0.16519 |
+--------------------------------------------------------------------------+-----------+ Each of these models accepts varied input formats and in Keras they have their own preprocessing functions. Benchmarking Outcome +----------------------------------+------------+----------------------+----------+
| Model Name | Parameters | Train time (seconds) | Log_Loss |
+----------------------------------+------------+----------------------+----------+
| EfficientNet_ConvBlock_Output | 54.7M | ~260s | 0.28347 |
| InceptionResNet_ConvBlock_Output | 56.1M | ~260s | 0.28623 |
| NASNetLarge_ConvBlock_Output | 89.6M | ~330s | 0.33285 |
| XCeption_ConvBlock_Output | 23.3M | ~240s | 0.34187 |
| InceptionV3_ConvBlock_Output | 24.2M | ~225s | 0.54297 |
| EfficientNet_GlobalAvg | 53.3M | ~260s | 0.50423 |
| InceptionV3_GlobalAvg | 22M | ~215s | 0.46402 |
| swin_base224 | 86.8M | ~550s | 0.47289 |
| swin_base384 | 87M | ~600s | 0.41902 |
| swin_large384 | 195M | ~1000s | 0.42207 |
+----------------------------------+------------+----------------------+----------+ SWIN Transformers have performed better than all of the ResNet50 variants and InceptionV3 model. Log-Loss of SWIN Transformer on this data is slightly higher compared to InceptionResNet, EfficientNet, Xception and NasNet Large models when their outputs are processed subsequently by convolutional layer followed by Maxpooling. SWIN however performs is as good as EfficientNet model when their average outputs are directly processed by the output layer. SWIN models are larger compared to any of the Convolutional models and hence will take a hit on the system throughput and latency. This study proved useful in understanding the application of transformer-based models for computer vision. You can find the code to the notebooks on my and the GUI for this usecase here. Github References: https://www.kaggle.com/competitions/dog-breed-identification https://arxiv.org/pdf/2103.14030.pdf https://www.youtube.com/watch?v=tFYxJZBAbE8&t=362s https://towardsdatascience.com/swin-vision-transformers-hacking-the-human-eye-4223ba9764c3 https://github.com/shkarupa-alex/tfswin

Image Classification in 2022

Clasificación de imágenes en 2022

Classification des images en 2022

2022 में छवि वर्गीकरण

2022 年の画像分類

Classificação de imagens em 2022

Phân loại hình ảnh năm 2022

2022年的图像分类

Open sourced by Google Research team, achieved wide popularity amongst NLP enthusiasts for all the right reasons! It is one of the best Natural Language Processing pre-trained models with superior NLP capabilities. It can be used for language classification, question & answering, next word prediction, tokenization, etc. pre-trained models of BERT Our case study Question Answering System in Python using BERT NLP and BERT based Question and Answering system demo , developed in Python + Flask, got hugely popular garnering hundreds of visitors per day. We got a lot of appreciative and lauding emails praising our QnA demo. Along with that, we also got number of people asking about how we created this QnA demo. And till the day, we keep getting requests on how to develop such a QnA system using BERT pre-trained model open-sourced by Google. [1] [2] To start with, the readme file on the official GitHub repository of BERT provides a good amount of information about how to fine-tune the model on SQuAD 2.0 but we could see that developers are still facing issues. So, we decided to publish a step-by-step tutorial to fine-tune the BERT pre-trained model and generate inference of answers from the given paragraph and questions on using TPU. Colab In this tutorial, we are not going to cover how to create web-based interface using Python + Flask. We’ll just cover the fine-tuning and inference on Colab using TPU. You can create your own interface using Flask or Django. Overview In this tutorial we will see how to perform a fine-tuning task on SQuAD using Google Colab, for that we will use BERT GitHub Repository, BERT Repository includes: TensorFlow code for the BERT model architecture. Pre-trained models for both the lowercase and cased version of BERT-Base and BERT-Large. You can also refer or copy to follow the steps. our colab file Steps to perform BERT Fine-tuning on Google Colab 1) Change Runtime to TPU On the main menu, click on and select . Set “ “ as the hardware accelerator. Below screeenshot will help you understand how you can change the runtime to TPU. Runtime Change runtime type TPU After Clicking on “Change runtime type”, Select from the dropdown option as given in the below figure. TPU 2) Clone the BERT github repository , or idirectional mbedding epresentations from ransformers, is a new method of pre-training language representations which obtains state-of-the-art results on a wide array of Natural Language Processing (NLP) tasks. You can find the academic paper here: . BERT B E R T https://arxiv.org/abs/1810.04805 BERT has two stages: and . Pre-training fine-tuning is fairly expensive (four days on 4 to 16 Cloud TPUs), but is a one-time procedure. BERT has released a number of pre-trained models. Most NLP researchers will never need to pre-train their own model from scratch. Pre-training is inexpensive. One can replicate all the results given in the paper, in at most 1 hour on a single Cloud TPU, or a few hours on a GPU. For example, SQuAD can be trained in around 30 minutes on a single Cloud TPU to achieve a Dev F1 score of 91.0%. Fine-tuning So our first step is to Clone the BERT github repository, below is the way by which you can clone the repo from github. Now get inside the Bert repo using “ “ command cd !git clone https: //github.com/google-research/bert.git cd bert 3) Download the BERT PRETRAINED MODEL BERT Pretrained Model List : BERT-Large, Uncased (Whole Word Masking): 24-layer, 1024-hidden, 16-heads, 340M parameters BERT-Large, Cased (Whole Word Masking): 24-layer, 1024-hidden, 16-heads, 340M parameters BERT-Base, Uncased: 12-layer, 768-hidden, 12-heads, 110M parameters BERT-Large, Uncased: 24-layer, 1024-hidden, 16-heads, 340M parameters BERT-Base, Cased: 12-layer, 768-hidden, 12-heads, 110M parameters BERT-Large, Cased: 24-layer, 1024-hidden, 16-heads, 340M parameters BERT-Base, Multilingual Cased (New, recommended): 104 languages, 12-layer, 768-hidden, 12-heads, 110M parameters BERT-Base, Multilingual Uncased (Orig, not recommended): 102 languages, 12-layer, 768-hidden, 12-heads, 110M parameters BERT-Base, Chinese: Chinese Simplified and Traditional, 12-layer, 768-hidden, 12-heads, 110M parameters BERT has released and models, that have uncased and cased version. Uncased means that the text is converted to lowercase before performing Workpiece tokenization, e.g., John Smith becomes john smith, on the other hand, cased means that the true case and accent markers are preserved. BERT-Base BERT-Large When using a cased model, make sure to pass -do_lower=False at the time of training. You can download any model of your choice. We have used the BERT-Large-Uncased Model. !wget https: //storage.googleapis.com/bert_models/2018_10_18/uncased_L-24_H-1024_A-16.zip # Unzip the pretrained model
!unzip uncased_L _H _A zip -24 -1024 -16. 4) Download the SQUAD2.0 Dataset For the Question Answering task, we will be using SQuAD2.0 Dataset. (Stanford Question Answering Dataset) is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable. SQuAD SQuAD2.0 combines the 100,000+ questions in SQuAD1.1 with over 50,000 new, unanswerable questions written adversarially by crowdworkers to look similar to answerable ones. You can download the dataset from SQUAD site https://rajpurkar.github.io/SQuAD-explorer/ #Download the SQUAD train and dev dataset
!wget https: !wget https: //rajpurkar.github.io/SQuAD-explorer/dataset/train-v2.0.json //rajpurkar.github.io/SQuAD-explorer/dataset/dev-v2.0.json 5) Set up your TPU environment Verify that you are connected to a TPU device You will get know your TPU Address which is used at the time of fine-tuning Perform Google Authentication to access your bucket Upload your credentials to TPU to access your GCS bucket Using code below you can do the above mentioned 4 points: datetime json os pprint random string sys tensorflow tf os.environ, TPU_ADDRESS = + os.environ[ ]
print( , TPU_ADDRESS) google.colab auth
auth.authenticate_user() tf.Session(TPU_ADDRESS) session:
  print( )
  pprint.pprint(session.list_devices()) open( , ) f:
    auth_info = json.load(f)
  tf.contrib.cloud.configure_gcs(session, credentials=auth_info) import import import import import import import import as assert 'COLAB_TPU_ADDR' in 'ERROR: Not connected to a TPU runtime; please see the first cell in this notebook for instructions!' 'grpc://' 'COLAB_TPU_ADDR' 'TPU address is => ' from import with as 'TPU devices:' # Upload credentials to TPU. with '/content/adc.json' 'r' as # Now credentials are set for all future sessions on this TPU. 6) Create an output directory : You will need a GCP (Google Compute Engine) account and a GCS (Google Cloud Storage) bucket to run the colab file. Please follow the Google Cloud for how to create a GCP account and GCS bucket. You have $300 free credit to start with any GCP product, learn more about it at . Prerequisite https://cloud.google.com/tpu/docs/setup-gcp-account You can create your GCS bucket from here . http://console.cloud.google.com/storage As we are using the Cloud TPU, we need to store the pre-trained model and the output directory in the Google Cloud Storage. If you are not storing it on Bucket you may face the following error : ERROR:tensorflow: recorded training_loop: From /job:worker/replica: /task: :
Unsuccessful TensorSliceReader : Failed to get matching files on uncased_L-24_H-1024_A-16/bert_model.ckpt: Unimplemented: File system scheme '[local]' not implemented (file: 'uncased_L-24_H-1024_A-16/bert_model.ckpt')
[[node checkpoint_initializer_14 (defined at /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py:1748) ]] Error from 0 0 constructor You will get your fine_tuned model in the Google cloud storage bucket after completion of training. For that, you need to provide your BUCKET name and OUPUT DIRECTORY name. BUCKET = BUCKET, output_dir_name = BUCKET_NAME = .format(BUCKET)
OUTPUT_DIR = .format(BUCKET,output_dir_name)
tf.gfile.MakeDirs(OUTPUT_DIR)
print( .format(OUTPUT_DIR)) 'bertnlpdemo' #@param {type:"string"} assert '*** Must specify an existing GCS bucket name ***' 'bert_output' #@param {type:"string"} 'gs://{}' 'gs://{}/{}' '***** Model output directory: {} *****' 7) Move Pretrained Model to GCS Bucket Need to move Pre-trained Model at GCS (Google Cloud Storage) bucket, as Local File System is not Supported on TPU. If you don’t move your pre-trained model to TPU you may face the error. The command allows you to move data between your local file system and the cloud, move data within the cloud, and move data between cloud storage providers. gsutil mv !gsutil mv /content/bert/uncased_L _H _A $BUCKET_NAME -24 -1024 -16 8) Training Below is the command to run the training. To run the training on TPU you need to make sure about below Hyperparameter, that is tpu must be true and provide the tpu_address that we have found above. -use_tpu=True -tpu_name=YOUR_TPU_ADDRESS !python run_squad.py \
  --vocab_file=$BUCKET_NAME/uncased_L _H _A /vocab.txt \
  --bert_config_file=$BUCKET_NAME/uncased_L _H _A /bert_config.json \
  --init_checkpoint=$BUCKET_NAME/uncased_L _H _A /bert_model.ckpt \
  --do_train= \
  --train_file=train-v2 .json \
  --do_predict= \
  --predict_file=dev-v2 .json \
  --train_batch_size= \
  --learning_rate= \
  --num_train_epochs= \
  --use_tpu= \
  --tpu_name=grpc:// : \
  --max_seq_length= \
  --doc_stride= \
  --version_2_with_negative= \
  --output_dir=$OUTPUT_DIR -24 -1024 -16 -24 -1024 -16 -24 -1024 -16 True .0 True .0 24 3e-5 2.0 True 10.1 .118 .82 8470 384 128 True Create Testing File We are creating input_file.json as a blank JSON file and then add the data in the file in the SQuAD dataset format. is used to create a file - is used to write a file in the colab - touch %%writefile You can pass your own questions and context in the below file. !touch input_file.json %%writefile input_file.json
{ : , : [
        { : , : [
                { : [
                        { : , : , : },
                        { : , : , : },
                        { : , : , : }
                    ], : }
            ]
        }
    ]
} "version" "v2.0" "data" "title" "your_title" "paragraphs" "qas" "question" "Who is current CEO?" "id" "56ddde6b9a695914005b9628" "is_impossible" "" "question" "Who founded google?" "id" "56ddde6b9a695914005b9629" "is_impossible" "" "question" "when did IPO take place?" "id" "56ddde6b9a695914005b962a" "is_impossible" "" "context" "Google was founded in 1998 by Larry Page and Sergey Brin while they were Ph.D. students at Stanford University in California. Together they own about 14 percent of its shares and control 56 percent of the stockholder voting power through supervoting stock. They incorporated Google as a privately held company on September 4, 1998. An initial public offering (IPO) took place on August 19, 2004, and Google moved to its headquarters in Mountain View, California, nicknamed the Googleplex. In August 2015, Google announced plans to reorganize its various interests as a conglomerate called Alphabet Inc. Google is Alphabet's leading subsidiary and will continue to be the umbrella company for Alphabet's Internet interests. Sundar Pichai was appointed CEO of Google, replacing Larry Page who became the CEO of Alphabet." Prediction Below is the command to perform your own custom prediction, that is you can change the input_file.json by providing your paragraph and questions after then execute the below command. !python run_squad.py \
  --vocab_file=$BUCKET_NAME/uncased_L _H _A /vocab.txt \
  --bert_config_file=$BUCKET_NAME/uncased_L _H _A /bert_config.json \
  --init_checkpoint=$OUTPUT_DIR/model.ckpt \
  --do_train= \
  --max_query_length= \
  --do_predict= \
  --predict_file=input_file.json \
  --predict_batch_size= \
  --n_best_size= \
  --max_seq_length= \
  --doc_stride= \
  --output_dir=output/ -24 -1024 -16 -24 -1024 -16 -10859 False 30 True 8 3 384 128 To make it easier for you, we have already created a Colab file which you can copy in your Google Drive and execute the commands. You can access the colab file at: . Question Answering System using BERT + SQuAD on Colab TPU If you have any further questions or doubt then please feel free to post them in comments. We’ll get back to you. Previously published at https://www.pragnakalp.com/nlp-tutorial-setup-question-answering-system-bert-squad-colab-tpu/ References Our case study Question Answering System in Python using BERT NLP BERT based Question and Answering system demo

NLP Tutorial: Creating Question Answering System using BERT + SQuAD on Colab TPU

What is ETL 💭? is short for , three database functions that are combined into one tool to pull data out of one database and place it into another database. ETL e xtract, t ransform, l oad is the process of from a database. In this stage, the data is collected, often from multiple and different types of sources. Extract reading data is the process of from its previous form into the form it needs to be in so that it can be placed into another database. Transformation occurs by using rules or lookup tables or by combining the data with other data. Transform converting the extracted data is the process of into the target database Load writing the data In the sample apps given by hasura, serverless-etl app is one among them. I will be showing how to implement it using Next.js . This is the to the example given by hasura. link The main features of this app will be :- Add Book to the database From the database data is saved in an algolia index Search for the book in the algolia index How will we implement it ? We will be using graphql mutations to insert book details into the database and when something is inserted into the database an event is triggered which will update the algolia index with the inserted data from the database and this data can be retrieved from the algolia index when searched. That’s it! it is that simple. Pre-requisites: Algolia account Hasura GraphQL Engine (HGE) installation Now that you are ready, Lets begin! Setting up basic Next.js project and Graphql Engine First step is to create a next.js project using . This command will create a basic boilerplate for a next.js project on which we will be working on. yarn create next-app etl-example Add the packages and scripts which are not present in your package.json file from the gist below and install it using . yarn install Next setup a Hasura GraphQL engine using the the deploy option and note down the deployed url and create a file in the root folder with the following code one-click config.js replace the with the url of the deployed hasura engine. Add authentication headers in the config object if you have any. herokuapp-url Creating Book Table We will be having a single table name which will have the details of the book like title and author. For that we have to go to the Hasura GraphQL engine console and create the following table book book table Setting up Algolia Index Next step is to setup an Algolia Index, For that follow the instructions given below. Sign-up for Algolia account Go to the Dashboard, Click on tab on the left sidebar Indices Click on button New Index Name it as demo_serverless_etl_app Goto tab on the sidebar API Keys Copy the (we'll call this ) Application ID **ALGOLIA_APP_ID** Click on tab and then click button All API Keys New API Key Give description as server key Choose the index demo_serverless_etl_app Select under the ACL section Add records Click and copy this API key from the list (we'll call it ) Generate **ALGOLIA_ADMIN_API_KEY** Setting up webhooks For adding the data into the algolia index when some data is inserted into the database, we will be needing a webhook which will be triggered on data insertion and this webhook will add the inserted data into the algolia index. For having a single api endpoint there is no point of having a seperate server. What we can do is setup a node server and add the api endpoint for the webhook and route all other url’s to the next.js handler. For the node server we have to create a file named in the root directory and copy the content from below. The route will be used to trigger an event when data is inserted into the table to add it into the algolia index. Add your algolia APP_ID and the ADMIN_API_KEY in the empty space present in the code. server.js /webhook Building the front-end Next we will setup our page where we will take in the author name and book title which will be added to the database and also a search bar where you can search for any book. Create a file named inside folder and add the following code and also replace the algolia api variables with your values. index.js post Search Component The component provided by package will do the search magic automatically. You do not have to worry about coding all those searching stuff 😉. The component will render the searchbox and component will render the results of your search InstantSearch react-instantsearch-dom SearchBox Hits Deploy Now that wehave the server ready, deploy it anywhere for example and you will get a url for the deployed server note that down, let it be <SERVER_URL>. glitch Creating an Event Trigger Our next step is to create an event trigger on adding a new book to the database, for this go to tab in Hasura Graphql Engine Console and add a new trigger. Replace with Events <TRIGGER_URL> <SERVER_URL>/webhook. Trigger Now you can open <SERVER_URL> in your browser and see the magic! 😉 And that’s it, we have the ETL-demo ready🙌 . I haven’t added any stylings in the code. If you want to make it look more awesome, just add the required css and make it better 😎. Link to the gists :- , , index.js server.js package.json About Me My name is . I am an undergraduate student at the Indian , pursuing my B.Tech degree in Information Technology. Aswin VB Institute of Information Technology, Allahabad I love creating and learning new things which i always do when i have free time. I love to code in J . avaScript and Python You can follow me on and . Twitter GitHub

How to implement ETL Application using Hasura GraphQL Engine And Next.js

I originally wrote this post for the . SocialCops engineering blog Photo by on arihant daga Unsplash In our , we mentioned how it has taken the data engineering ecosystem by storm. We also talked about how we’ve been using it to move data across our internal systems and explained the steps we took to create an internal workflow. The ETL workflow (e)xtracted PDFs from a website, (t)ransformed them into CSVs and (l)oaded the CSVs into a store. We also touched briefly on the breadth of ETL use cases you can solve for, using the Airflow platform. last post on Apache Airflow In this post, we will talk about how one of Airflow’s principles, of being ‘Dynamic’, offers as a powerful construct to automate workflow generation. We’ll also talk about how that helped us use Airflow to power DISHA, a national data platform where Indian and monitor the progress of 42 national level schemes. In the end, we will discuss briefly some of our reflections from the project on today’s public data technology. configuration-as-code MPs MLAs Why Airflow? To recap from the , Airflow is a workflow management platform created by Maxime Beauchemin at Airbnb. We have been using Airflow to set up batching data workflows in production for more than a year, during which we have found the following points, some of which are also its core principles, to be very useful. previous post : A workflow can be defined as a (DAG) in a Python file (the DAG file), making dynamic generation of complex workflows possible. Dynamic Directed Acyclic Graph An Airflow DAG : There are a lot of operators right out of the box! An operator is a building block for your workflow and each one performs a certain function. For example, the PythonOperator lets you define the logic that runs inside each of the tasks in your workflow, using Python! Extensible : The tasks in your workflow can be executed parallely by multiple Celery workers, using the CeleryExecutor. Scalable : The project is under incubation at the Apache Software Foundation and being actively maintained. It also has an active . Open Source Gitter room Furthermore, Airflow comes with a web interface that gives you all the context you need about your workflow’s execution, from each task’s state (running, success, failed, etc.) to logs that the task generated! The problem with static code Here at SocialCops, we’ve observed a recurring use case of extracting data from various systems using web services, as a component of our ETL workflows. One of the ways to go forward with this task is to write Python code, which can be used with the PythonOperator to integrate the data into a workflow. Let’s look at a very rudimentary DAG file that illustrates this. As you can observe, the can be instantiated by specifying the name of the function containing your Python code using the keyword argument. Multiple instantiated operators can then be linked using Airflow API’s and methods. PythonOperator python_callable set_downstream set_upstream In the DAG file above, the function makes a GET request to , with a query parameter. Web services can vary in their request limit (if they support multiple requests at the same time), query parameters, response format and so on. Since writing custom Python code for each web service would be a nightmare for anyone maintaining the code, we decided to build a Python library (we call it Magneton, since it is a magnet for data), which takes in the JSON configuration describing a particular web service as input and fetches the data using a set of pre-defined queries. But that solved only half of our problem. extract httpbin.org In our and in the example DAG file above, we could link operators together by writing static calls to the and methods since the workflows were pretty basic. But imagine a DAG file’s readability with 1,000 operators defined in it. You would have to be a savant to infer the relationships between operators. Moreover, everyone in your team (including people who don’t work with Python as their primary language) wouldn’t have the know-how to write a DAG file, and writing them manually would be repetitive and inefficient. last post set_downstream set_upstream Meta data engineering In his talk “ “, Maxime Beauchemin, the author of Airflow, explains how data engineers should find common patterns (ways to build workflows dynamically) in their work and build frameworks and services around them. He gives some examples of such patterns, one of which is AutoDAG. It is an Airflow plugin that Airbnb developed internally to automate DAG creation, allowing the users who just need to run a scheduled SQL query (and don’t want to go author a workflow) to create the query on the web interface itself. Advance Data Engineering Patterns with Apache Airflow Finding patterns involves identifying the building blocks of a workflow and chaining them based on a static configuration. Look at the DAG file that we showed in the section above and try to identify the building blocks. It has just three components, which can be modeled into a configuration. YAML This is a very basic example. For a more detailed one, you should check out how and have modeled their workflow specifications, which they use to dynamically generate workflows. Pachyderm Digdag This makes it easy for us to now write a DAG file that can take in a bunch of these YAML configurations and build DAGs dynamically, by linking operators which have the same identifiers (in this example, we have used a number, 1, for the sake of simplicity). Moreover, anyone in your team who wants to create a workflow can just write a YAML, which makes it easy for a human to define a configuration that is machine-readable. Once you’ve figured out a way to create DAGs based on configurations, you can build a interface to let users build a DAG without writing configurations, making it easy for anyone looking to create a workflow! single For DISHA, we needed to (E)xtract scheme data from source systems via web services and then follow that with the T and L. At an atomic level, our workflows could be broken down into: a PythonOperator to (E)xtract data from the source system, by using Magneton (the Python library we had developed). a BashOperator to run R or Python-based (T)ransformations on the extracted data, like cleaning, reshaping, and standardizing geographies across data sets. a PythonOperator to (L)oad the transformed data into our data warehouse, on which can run analytical queries. Visualize Additionally, we added Slack and email alerts using the PythonOperator. Identifying this pattern let us automate DAG creation. Using a web interface, anyone could now add the configuration needed for the three basic tasks outlined above. This helped us to distribute the work of setting up workflows within our small team, most of whom were comfortable only in R. Soon, everyone was writing R scripts and building intricate workflows, like the one below. The Airflow web interface lets the project stakeholders manage complex workflows (like the one shown above) with ease, since they can check the workflow’s state and pinpoint the exact step where something failed, look at the logs for the failed task, resolve the issue and then resume the workflow by retrying the failed task. Making tasks is a good practice to deal with retries. (Note: retries can be automated within Airflow too.) idempotent An overview on DISHA Narendra Modi, Prime Minister of India ‘DISHA is a crucial step towards good governance through which we will be able to monitor everything centrally. It will enable us to effectively monitor every village of the country.’ — The District Development Coordination and Monitoring Committee (DISHA) was . The goal was to coordinate between the Central, State and Local Panchayat Governments for successful and timely implementation of key schemes (such as the National Rural Livelihoods Mission, Pradhan Mantri Awaas Yojana and Swachh Bharat Mission). To monitor the schemes and make data-driven implementation decisions, stakeholders needed to get meaningful insights about the schemes. This required integrating the different systems containing the scheme data. formed in 2016 Last year, we partnered with the Ministry of Rural Development (MoRD) and National Informatics Centre (NIC) to create the DISHA dashboard, which was . The DISHA Dashboard helps Members of Parliament (MPs), Members of Legislative Assembly (MLAs) and District Officials track the performance of flagship schemes of different central ministries in their respective districts and constituencies. launched by the Prime Minister in October Back in October 2017, the dashboard had data for 6 schemes, and it was updated in August 2018 to show data for a total of 22 schemes. In its final phase, the dashboard will unify data from 42 flagship schemes to help stakeholders find the answer to life, universe and everything. For the first time, data from 20 ministries will break silos to come together in one place, bringing accountability to a government budget of over Rs. 2 lakh crores! The DISHA meetings are held , where the committee members meet to ensure that all schemes are being implemented in accordance with the guidelines, look into irregularities with respect to implementation and closely review the flow of allocated funds. Workflows, like the one showed above, have automated the flow of data from scheme databases to the DISHA Dashboard, updating the dashboard regularly with the most recent data for a scheme. This is useful for the committee members since they can plan for the meeting agenda by checking each scheme’s performance and identifying priorities and gap areas. once every quarter Watch the Prime Minister speak about how he uses the DISHA dashboard to monitor the progress of Pradhan Mantri Awas Yojna here . Going towards better public technology Data extraction is a piece of a larger puzzle called (getting the data you want to a single place, from different systems, the way you want it), which people have been working on since the . Integrating different data systems can be quite complex due to the systems being heterogeneous; this means they can differ in the way they are implemented, how data is stored within them and how they interact with other systems, making them silos of information. data integration early 1980s To successfully extract data from another system, people on both ends of the transaction first need to agree upon a schema for how the data will be shared. As we found, this can be the most time-consuming part of a project, given how heterogeneous different data systems can be. Even though we have successfully integrated 22 data sources together so far, the time we spent on getting the data in the right format, with the variables we needed for each scheme, would’ve been saved if there was a standard for storing and sharing data across all ministries. In his article “ ”, Varun Adibhatla of talks about ‘Information Exchange protocols’. He calls it jargon for, being able to share standardized data or speak a common language at some considerable scale. He further explains that, in the 1990s, Wall Street financial institutions got together and agreed to speak in a common transactional language. This led to the creation of the FIX protocol, which let them share data quickly. He mentions that data standards like the FIX protocol are not unique to Wall Street, but exist in almost every sphere of trade and commerce. Bringing Wall Street and the US Army to bear on public service delivery ARGO Labs here Varun Adibhatla in ‘A small team of purpose-driven public technologists, leveraging advances in low-cost device, data and decision-making and the right kind of support is all it takes to build and maintain public, digital infrastructures.’ — “Bringing Wall Street and the US Army to bear on public service delivery.” As we move towards a Digital India, we need a fundamental shift from the Excel-for-everything mindset and how today’s public technology is set up. We need a standardized data infrastructure across public services that will help ministries and departments share data with each other quickly, and with the public. A with Silicon Valley–grade technical chops need to be trained and hired. There’s already a Chief Economic Advisor to the government and a Chief Financial Officer for the RBI. It’s high time a Chief Data Officer is appointed for India! new generation of public servants

Airflow, Meta Data Engineering, and a Data Platform for the World’s Largest Democracy

I originally wrote this post for the . SocialCops engineering blog What is the first thing that comes to your mind upon hearing the word ‘Airflow’? Data engineering, right? For good reason, I suppose. You are likely to find Airflow mentioned in every other blog post that talks about data engineering. is a workflow management platform. To oversimplify, you can think of it as cron, but on steroids! It was started in October 2014 by Maxime Beauchemin at Airbnb. From the very first commit, Airflow was open source. Less than a year later, it was moved into the Airbnb Github. Since then, it has become a vital part of the data engineering ecosystem. Apache Airflow We have been using Airflow to move data across our internal systems for more than a year, over the course of which we have created a lot of ETL (Extract-Transform-Load) pipelines. In this post, we’ll talk about one of these pipelines in detail and show you the set-up steps. Note: We will not be going through how to set up Airflow. You can check out a great blog from Clairvoyant for that. Why use Airflow? A workflow can be defined as a Directed Acyclic Graph (DAG). Airflow will make sure that the defined tasks are executed one after the other, managing the dependencies between tasks. Dependency Management: : Airflow offers a variety of Operators, which are the building blocks of a workflow. One example is the PythonOperator, which you can use to write custom Python code that will run as a part of your workflow. Extensible : Celery, which is a distributed task queue, can be used as an Executor to scale your workflow’s execution. Scalable : It is under incubation at the Apache Software Foundation, which means it is being actively maintained. Open Source IDSP: The disease data source Even though open data portals are cropping up across multiple domains, working with the datasets they provide is difficult. In our bid to identify and help prevent disease outbreaks at , we came across one such difficult data source. SocialCops The (MHRD) runs the (IDSP) scheme, which identifies disease outbreaks at the sub-district & village level across India. Under this scheme, the MHRD releases weekly outbreak data as a PDF document. Ministry of Health and Family Affairs Integrated Disease Surveillance Programme PDFs are notorious for being hard to scrape and incorporate in data science workflows, but just wait till you see the IDSP PDFs. It may look like the data in them is in a nice table format, but they’ve changed the table formatting over the years and may continue to do so. We’ve also encountered glitches in the document like different tables being joined together, tables flowing out of the page and even tables within tables! Setting up the ETL pipeline No brownie points for figuring out the steps involved in our pipeline. We ( )xtract the PDFs from the IDSP website, ( )ransform the PDFs into CSVs and ( )oad this CSV data into a store. E T L Conventions Let us set up some conventions now, because without order, anarchy would ensue! Each Directed Acyclic Graph should have a unique identifier. We can use an ID, which describes what our DAG is doing, plus a version number. Let us name our DAG idsp_v1. Note: We borrowed this naming convention from the Airflow “Common Pitfalls” documentation . It comes in handy when you have to change the start date and schedule interval of a DAG, while preserving the scheduling history of the old version. Make sure you check out this link for other common pitfalls. We will also define a base directory where data from all the DagRuns will be kept. What is a DagRun, you ask? It is just an instance of your DAG in time. We will also create a new directory for each DagRun. Here’s a requirements.txt file which you can use to install the dependencies. How to DAG In Airflow, DAGs are defined as Python files. They have to be placed inside the , which you can define in the Airflow configuration file. Based on the ETL steps we defined above, let’s create our DAG. dag_folder We will define three tasks using the Airflow PythonOperator. You need to pass your Python functions containing the task logic to each Operator using the keyword argument. Define these as dummy functions in a file for now. We’ll look at each one later. python_callable utils.py We will also link them together using the methods. This will define the order in which our tasks get executed. Observe how we haven’t defined the logic that will run inside the tasks, but our DAG is ready to run! set_downstream Have a look at the DAG file. We have set the to . Yes, you guessed it correctly — it’s a cron string.This means that our DAG will run every Tuesday at 12 AM. Airflow scheduling can be a bit confusing, so we suggest you check out the to understand how it works. schedule_interval 0 0 * * 2 Airflow docs We have also set to since we want Airflow to pass the DagRun’s context (think metadata, like the dag_id, execution_date etc.) into our task functions as keyword arguments. provide_context True Note: We’ll use execution_date (which is a Python datetime object) from the context Airflow passes into our function to create a new directory, like we discussed above, to store the DagRun’s data. At this point, you can go ahead and create a DAG run by executing on the command line. Make sure you go to the Airflow UI and unpause the DAG before creating a DagRun. The DagRun should be a success since our tasks are just dummy functions. airflow trigger_dag idsp_v1 Now that we have our DAG file ready, let’s look at the logic that will run inside our tasks. Note: Everything you print to standard output inside the function passed to the PythonOperator will be viewable on the Airflow UI. Just click on View Log in the respective operator’s DAG node. Scraping the IDSP website A new PDF document is released almost every week (with some lag) on the IDSP website. We can’t keep scraping all the PDFs every time a new one is released. Instead, we will have to save the week number of the PDF we last scraped somewhere. We can store this state in a CSV file in our base directory at the end of each DagRun and refer to it at the start of another. Take a look at the scraping code. nThere’s nothing fancy here, just your run-of-the-mill web scraping, using and . requests lxml Note: In production, we don’t run the scraping code inside Airflow. It is run on a separate service, which can connect to REST/SOAP APIs to extract data, in addition to running these scrapers. This gives us a central place to schedule and track how data is pulled into our platform. The task logic is replaced with a call to the data export service. Scraping the PDFs Yay! Now that we have new PDFs, we can go about scraping them. We will use to accomplish this. pdfminer But first, let me just point out that PDF is the worst format for tabular data. A PDF contains instructions for PDF viewers to place text in the desired font at specific X,Y coordinates on the 2D plane. It doesn’t matter if we just need to get text from of a PDF, but if we need to get tabular data with the table structure intact, it gets difficult. We use a simple heuristic here to get this data out. First, we extract all text objects using . A text object contains a string and a float coordinate tuple, which describes the string’s position. We then sort all these objects in increasing X and increasing Y order. We now have all these text objects in row-major format, starting from the last row. We can just group them based on their x-axis projections into different columns, discarding any objects that span multiple columns. We can drop the first and last columns since we don’t have any use for them in this post. pdfminer Voila, we have our table! It is still not usable though and needs some minor cleaning. We are leaving this as an exercise to you. It is easy to define some rules in code to convert the above CSV to something cleaner, like the following. You can add this cleaning code as another PythonOperator or within the same operator. If you are not comfortable with Python and want to use R instead, you can use the BashOperator to call your R script. Extensibility FTW! scrape_pdf Note: It is very difficult to get 100% table scraping accuracy on all types of PDF with a single tool. We can just throw various heuristics at the problem and hope for the best result. A cleaning step is usually required. When we were in the process of preparing the IDSP dataset, using all the previous years’ PDFs, we couldn’t find any tool/library that could solve this problem. We tried many open source tools like Tabula , as well as closed source tools like PDFTables without any success. This led us to developing our own library, which uses image recognition with a bunch of heuristics to try and solve the PDF table scraping problem. It gave us an acceptable scraping accuracy on a lot of PDF types, including the IDSP ones. Once we plugged this into our data cleaning product, Transform , we could finally convert PDF data into a fully clean CSV. Update (5th October 2018): We released Camelot, a Python library that helps anyone extract tabular data from PDFs . You can find a version of the code provided in this blog post that uses Camelot in this Jupyter notebook. Curating the scraped data Now that we have a clean CSV, we can add it to our master IDSP dataset. The operator contains just a for loop, which appends page-wise CSVs to our master CSV dataset. We could’ve used here, but we didn’t want to add another requirement just for this append. pandas Internally, our ETL pipeline doesn’t stop here though. We pass the text in the ‘Comments’ column that we dropped earlier through our entity recognition system, which gives us a list of geographies where the outbreaks happened. This is then used to send alerts to our team and clients. Where can you go from here? Congrats! You have a regularly updating disease outbreaks data set! Now it’s up to you to figure out how you’re gonna use it. predictive analytics You can replace the scraping code to scrape data from any other website, write it to the run directory, plug in the PDF scraping operator (if the data you scraped is in PDF format), or plug in a bunch of your own operators to do anything. cough cough. You can find the complete code repo for this exercise here. If you do extend this DAG, do . We’d love to hear what you did! tweet at us Seize the data!

FANSFORX

fansforx.com

51-200 employees

Since 2017

COMPANY RANKING

FANSFORX

EVERGREEN INDEX #4032

Representing this company? Click here to claim and customize this page!

FansForX WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

FANSFORX

fansforx.com

51-200 employees

Since 2017

COMPANY RANKING

FANSFORX

EVERGREEN INDEX #4032

RELATED COMPANIESRANK

STORIES ABOUT FANSFORX

Representing this company? Click here to claim and customize this page!

FansForX WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!