Daato

daato

Daytona, a pioneering development environment management (DEM) platform offering secure and scalable hybrid development environments, just closed a $2 million pre-seed funding round. I've spoken with CEO and co-founder to learn more about the investment, Daytona's traction, and future plans. Ivan Burazin How much money did you raise in this round? We raised $2 million, which exceeded our initial target. There was tremendous interest from leading dev tools investors who recognized the potential of our solution. Who participated in the round? When we structured our Pre-Seed round, we intentionally sought support from experienced founders and key players in the developer tool industry, individuals whose reputations are linked to the tools you probably use daily. We received backing from angel investors like Abhinav Asthana (Founder of ), Prashanth Chandrasekar (CEO of ), Charity Majors (Co-Founder of ), Paul Copplestone (Co-Founder of ), Luke Kanies (Founder of ) and many others. Their domain expertise will help us refine our product and scale efficiently. Postman StackOverflow Honeycomb Supabase Puppet To bolster our capacity to meet growing demand, we also teamed up with Emerging Europe, a firm managing $2.7 billion in assets and an investor in over 35 billion-dollar enterprises such as GitLab, Twilio, Algolia, among others. Additionally, we're excited to reveal involvement from , , and , all of whom have supported top-tier companies, including Anthropic, Stripe, and OpenAI. 500 Tiny.vc Silicon Gardens Firestreak Ventures One of our angels, Christian Bach from , was impressed by our dedication to improving developer environments. He believes in our mission to make building more straightforward, a mission that's in line with the larger aim of boosting developer efficiency. Netlify standardized, secure developer environments What is your company’s current valuation? The round was done on an SAFE, reflecting the immense market potential for secure, self-hosted development environments tailored to enterprise needs. Our early traction with large Fortune 500 customers validates the strong demand. uncapped What does Daytona offer today? Daytona provides a self-hosted that gives enterprises an enterprise-grade alternative to GitHub Codespaces. Developers can spin up secure, compliant coding environments matching production infrastructure with a simple CLI command or click in the Daytona dashboard. development environment management platform Daytona eliminates complex configuration and security challenges while empowering developers with flexible, instantly available workspaces. We aim to be the industry standard for dev environments served through internal developer platforms. How do you differentiate from competitors? Unlike other development environment managers, Daytona is designed for self-managed deployment on internal infrastructure, whether cloud or local data center. This enables complete data control, security, and policy enforcement for regulated industries. Who are the founders and key team members? My co-founders, (CTO) and (Chief Architect), and I lead an experienced team of cloud dev tools veterans. We have pioneered the cloud IDE space with and have over 15 years of experience understanding developer needs. Building on this experience, Daytona offers unmatched flexibility, security, and simplicity. Vedran Jukic Goran Draganic Codeanywhere Our technical leadership designed Daytona's architecture leveraging Kubernetes, Kata Containers, and other leading orchestration technologies. We're a product-obsessed team focused on delivering exceptional solutions. How will you invest the funding? The funding will help us accelerate hiring across engineering, sales, and marketing to keep pace with customer demand. We'll also be refining our product capabilities, exploring strategic partnerships, and ramping up sales and marketing efforts. What’s exciting about your traction? We already have large enterprise customers in industries like aerospace, banking, and telecommunications. Our early success in securing Fortune 500 customers indicates the immense market appetite for enterprise-tailored development environments. What are your revenue goals? As this is an early stage, we are focused on product-market fit and scaling our customer base. In the forthcoming period, our goal is to double our revenue. Our vision is ambitious but achievable based on the enthusiasm we're seeing. What's your favorite pitch deck slide? My favorite slide captures the all-too-familiar pre-coding routine developers endure. It illustrates the steps before actual coding can begin, such as switching git branches, refreshing local environments, downloading libraries, installing tools, enduring build times, updating dependencies, compiling the project, and waiting for tests to run. When scaled across multiple projects and teams, this sequence of tasks significantly amplifies the inefficiency and frustration. This slide resonates deeply with our audience because it highlights the universal pain points in the development process, and it sets the stage for Daytona to present our compelling solution. It effectively underscores the necessity for a system like ours, which streamlines these cumbersome processes, allowing developers to focus on what they do best—coding. What problem does Daytona solve? Daytona addresses the challenges enterprises face in building secure, compliant development environments cost-effectively. We eliminate the need for expensive in-house platforms by providing an off-the-shelf solution purpose-built for regulated industries. The rise of infrastructure-as-code and integrated IDEs created the perfect timing for our standardized environment approach. We make robust dev environments accessible. What are the biggest risks? There is always execution risk with bringing a new product to market. However, our experience, technical expertise, customer validation, and exceptional investor support give us confidence. Our biggest challenge is keeping up with surging enterprise demand. We'll be smart about scaling sustainably. Why will Daytona grow exponentially? Every software team needs standardized environments to eliminate configuration drift and boost velocity. We have a track record as founders who helped pioneer cloud-based dev tools. With tailored enterprise capabilities, incredible support, and perfect timing, Daytona is poised for tremendous growth. The market pain points are clear, the solution works, and customers are validating the value. We enable enterprises to deliver better software faster through Daytona's simplicity and security. It's a very exciting time!

Interview with Daytona CEO Ivan Burazin on $2M Pre-Seed Round

The Noonification: DataOps: the Future of Data Engineering (9/17/2023)

The Noonification: DataOps: the Future of Data Engineering (9/9/2023)

In today's dynamic digital landscape, data is frequently dubbed as the new oil. However, as with crude oil, the true value of data lies not in its raw state but in its refinement: processing, analyzing, and leveraging it to inform decisions. and Data Engineering often intertwine, especially as DevOps principles are increasingly applied in the data domain, giving rise to the term "DataOps." As data volumes have surged and the pace of business has intensified, the methodologies used to manage and harness data have consequently evolved. Enter DataOps. DevOps Introduction to DataOps At its core, DataOps can be understood as the application of DevOps principles to data workflows. If DevOps is about enhancing software development through continuous delivery, integration, and collaboration, DataOps takes a leaf out of that book but specifically targets the unique challenges and intricacies of the data domain. The focus here is not just on the data itself but on the processes, systems, and teams that handle it, ensuring a streamlined flow of data through its lifecycle. Definition: The concept of DataOps didn't emerge in isolation. It was born out of a confluence of needs. With businesses increasingly relying on data-driven insights and with data sources becoming more varied and voluminous, traditional data management practices began to show their limitations. Speed, scalability, and collaboration became paramount. DataOps, as an approach, traces its roots back to these industry demands and the successful paradigms established by DevOps. Over time, as the industry started recognizing the gaps between data teams (from engineers to scientists to analysts) and the operational challenges they faced, DataOps began to solidify as a distinct discipline in data management. Evolution: Parallels between DevOps and DataOps The continuous evolution in technology and business demands has brought about various methodologies and practices. While DevOps primarily emerged as a bridge between software development and IT operations, ensuring swift and efficient software releases, its foundational principles found resonance in another crucial domain: data engineering. This synergy paved the way for DataOps. To grasp the essence of DataOps, one must first understand its parallels with DevOps. Shared Principles: Just as DevOps emphasizes the and infrastructure changes to ensure consistent and rapid delivery, DataOps recognizes the need for automating data pipelines. This automation reduces manual intervention, minimizes errors, and accelerates data processing. For instance, where DevOps might automate code testing and deployment, DataOps automates data validation, transformation, and loading processes. Imagine a global e-commerce platform. Where DevOps might ensure that the platform itself remains bug-free and user-friendly through automated testing, DataOps ensures that when a customer views a product, the stock count, price, and other product details are up-to-date and accurate, achieved through automated data validation and loading. Automation: automation of software deployment The heart of DevOps lies in the , which ensures that code changes are continuously integrated, tested, and delivered to production. Similarly, DataOps adopts CI/CD but in the realm of data. It ensures that data from various sources integrates seamlessly, is consistently refined, and is delivered to data warehouses or analytics tools without hiccups. This continuous flow ensures that businesses always have access to timely and reliable insights. Continuous Integration and Continuous Delivery (CI/CD): CI/CD pipeline DevOps introduced a culture where developers and IT operations teams collaborated closely, breaking down silos. DataOps extends this collaborative approach to include data engineers, data scientists, and business stakeholders. The objective is the same: fostering an environment where cross-functional collaboration results in optimal outcomes, whether it's software products in DevOps or data-driven insights in DataOps. For instance, a tech company launching a new device might need both software updates (managed by DevOps) and real-time market data for pricing and promotion (handled by DataOps). Seamless collaboration between these teams can ensure a successful launch. Collaboration: Customization for Data: While the foundational principles might be shared, it's essential to understand that data has its own set of complexities. This is where DataOps customizes these principles. Unlike code repositories, which are relatively standardized, data can come from a plethora of sources: IoT devices, user interactions, databases, third-party APIs, and more. DataOps ensures that the principle of continuous integration is agile enough to handle this diversity, integrating data from varied sources into a cohesive dataset. Diverse Data Sources: Data isn't always clean or reliable. Unlike software bugs that are generally consistent in nature, data anomalies can be random and multifaceted. DataOps, drawing from the automation principle, incorporates automated data quality checks and validation processes, ensuring that the data utilized is accurate and trustworthy. Data Quality and Integrity: Data volumes are ever-growing. While DevOps deals with the scalability of applications and infrastructure, DataOps must ensure that and pipelines scale efficiently, processing ever-larger datasets without performance bottlenecks. Scalability and Performance: data infrastructures In essence, DataOps is not just a mere adoption of DevOps principles but a meticulous adaptation, ensuring these principles serve the unique demands and challenges of the data world. It's this careful melding of principles and customizations that makes DataOps a vital methodology in today's data-driven enterprises. Importance of DataOps In the evolving landscape of data-driven decision-making, organizations are under constant pressure to efficiently utilize their data resources. With the sheer volume and diversity of data at play, traditional data management techniques often fall short of delivering timely insights. DataOps, as a solution, steps into this arena, emphasizing not just the methods but the value it brings to the table. The importance of DataOps is manifold, addressing time-to-value, collaborative efficiency, and data quality with remarkable efficacy. Time-to-Value: In the world of business, time is money. The quicker raw data is converted into actionable insights, the faster organizations can make informed decisions, be it launching a new product, refining marketing strategies, or identifying operational inefficiencies. DataOps plays a pivotal role here: Streamlined Workflows: By applying continuous integration and delivery principles, DataOps ensures that data pipelines are smooth, reducing the lag between data acquisition and data consumption. Data validation, transformation, and loading are automated, minimizing manual interventions that often lead to delays. Automated Processes: Airlines use DataOps to quickly process vast amounts of flight data, weather conditions, and passenger preferences to optimize flight schedules, pricing, and in-flight services. This immediate processing can lead to improved customer experience and efficient operations. Collaborative Efficiency: The true potential of data is realized when diverse teams —including data engineers, data scientists, and business stakeholders—work in unison. DataOps fosters this synergy. DataOps encourages the use of platforms where teams can collaboratively view, access, and work on data. This shared workspace reduces back-and-forths and promotes parallel processing. Unified Data Platforms: With clear communication channels, teams are better aligned in their goals, ensuring that the data engineering process serves the larger business objectives. Shared Objectives: Data Quality and Reliability: High-quality insights require high-quality data. With the vastness and varied nature of data sources, ensuring data consistency and reliability is paramount. This is where DataOps shines: Just as code undergoes tests in DevOps, DataOps incorporates automatic data quality assessments, ensuring anomalies are detected and rectified early in the pipeline. Consider financial institutions like , where trading algorithms are based on vast amounts of data. It's not just about quantity; the quality of this data is paramount. A single discrepancy could lead to significant financial discrepancies. Through DataOps' automated quality checks, the bank ensures that its algorithms operate on accurate, validated data, reducing potential risks. Automated Quality Checks: JP Morgan Borrowing from DevOps, DataOps often uses version control for data, ensuring that every stakeholder accesses the most recent, consistent version of the dataset. Version Control: Feedback Loops: Continuous monitoring of data pipelines means that any discrepancies in data quality are flagged and fed back into the system for improvements. This iterative approach enhances the reliability of the data over time. In the midst of an information age where data is plentiful but actionable insights are gold, DataOps stands as a beacon, guiding organizations to effectively harness their data potential. By focusing on quick turnarounds, collaboration, and quality, DataOps ensures that data engineering is not just about managing data but truly empowering businesses. Challenges Addressed by DataOps In today's digital era, businesses are inundated with data. Yet, while data is undeniably a valuable asset, it comes with its own set of challenges. These challenges, if not addressed, can hinder an organization's ability to make informed decisions, strategize effectively, and maintain a competitive edge. DataOps, as a forward-thinking methodology, aims to alleviate these pain points. Let's delve into the core challenges DataOps addresses: Scalability: With data streaming in from a myriad of sources, data infrastructures often buckle under the strain of ever-expanding data volumes. Traditional systems might be ill-equipped to handle this influx, leading to performance degradation and bottlenecks. DataOps encourages the use of cloud-based solutions and containerization, allowing data infrastructures to dynamically scale as per the data volume. This ensures smooth and efficient data processing irrespective of the data load. For instance, Netflix, known for its massive user base, leverages DataOps principles to handle petabytes of data daily, ensuring that their recommendation algorithms and content delivery networks run efficiently. Dynamic Scaling: Through continuous monitoring and feedback loops, DataOps ensures that resources are optimally utilized, preventing over-provisioning and wastage. Resource Optimization: Data Variety: Data heterogeneity poses another challenge. Organizations handle data that ranges from structured datasets in relational databases to unstructured data from social media, logs, and IoT devices. DataOps promotes the creation of platforms where diverse datasets can be integrated, transformed, and standardized, thus offering a cohesive view of data. Coca-Cola, for example, integrates data from various sources such as sales figures, social media feedback, and supply chain data using DataOps techniques to gain a holistic view of its global operations. Unified Data Platforms: DataOps practices often emphasize robust metadata management, aiding in understanding, categorizing, and utilizing diverse datasets effectively. Metadata Management: Real-time Processing: In an age where real-time insights can be a game-changer, the demand for real-time data processing has surged. This requires handling data streams efficiently and processing them without delays. DataOps ensures that data pipelines are designed for low latency, handling data streams efficiently and delivering real-time insights. utilize DataOps to process real-time data on traffic, driver availability, and user demand to efficiently match drivers with riders. Streamlined Pipelines: Companies like Uber DataOps often leans on event-driven architectures, which respond in real-time to data changes or specific events, ensuring timely data processing and analytics. Event-Driven Architectures: Data Security and Compliance: With data breaches becoming increasingly common and regulations like GDPR in place, data security and compliance cannot be overlooked. DataOps emphasizes encrypting data at rest and in transit, ensuring that sensitive information is always secure. End-to-End Encryption: With tools integrated into the DataOps workflow, compliance checks are automated, ensuring data handling adheres to regulatory standards consistently. Automated Compliance Checks: Access Control: Role-based access controls are instituted, making sure that only authorized personnel can access and modify sensitive data. In addressing these challenges, DataOps transforms the daunting task of data management into a streamlined, efficient, and secure process. Organizations armed with DataOps are better positioned to harness their data's potential, ensuring that data-related challenges are not roadblocks but mere stepping stones toward a data-driven future. Best Practices for Implementing DataOps Implementing DataOps can significantly refine an organization's data engineering processes, but the effectiveness of this methodology hinges on the adoption of best practices. These practices act as guideposts, ensuring that DataOps not only integrates smoothly into the existing data ecosystem but also realizes its transformative potential. Here's a comprehensive look at these best practices: Setting Objectives: Defining the North Star: Before diving into DataOps, organizations must outline their primary goals. Whether it’s streamlining data processing, bolstering data quality, or fostering inter-departmental collaboration, a well-articulated objective is pivotal. Companies like Airbnb have emphasized the importance of setting clear objectives when transitioning to DataOps, which allowed them to streamline their vast property and user data more efficiently. Key Performance Indicators (KPIs) act as tangible metrics of success. By setting quantifiable targets, organizations can gauge the efficacy of their DataOps practices and iteratively refine their approach. Without a set target, many organizations find themselves adrift in the vast sea of data, leading to wasted resources and missed opportunities. Quantifying Success: Building the Right Team: The essence of DataOps lies in interdisciplinary teamwork. Assembling a diverse group comprising data scientists, engineers, and operations experts ensures a holistic approach to data challenges. Tech giants like Google stress the importance of diverse teams, drawing on varied expertise to tackle complex data scenarios. Cross-functional Collaboration: The fluid nature of data mandates that the team remains updated with prevailing trends and methodologies. Regular workshops and training sessions help maintain a cutting-edge team. Moreover, fostering soft skills like adaptability, effective communication, and problem-solving enhances the team's ability to navigate the challenges of DataOps efficiently. Continuous Training: Choosing the Right Tools: The market is inundated with tools designed for containerization, orchestration, version control, and monitoring. Organizations should carefully assess their requirements, conduct pilot tests, and choose tools that align with their objectives and infrastructure. The plethora of available tools can be overwhelming. Prioritizing tools like Apache Airflow for orchestration or Docker for containerization, after meticulous assessment, can be beneficial. It’s also prudent to be wary of tools that promise the moon but might not align with the organization's specific needs or existing infrastructure. Assessment and Evaluation: The chosen tools should seamlessly integrate with existing systems, ensuring that the transition to a DataOps approach is smooth and devoid of disruptions. Integration Capabilities: Promoting a DataOps Culture: Collaboration is the bedrock of DataOps. Creating a milieu where open dialogue is the norm and where teams from diverse domains unite to solve data challenges is paramount. Organizations that sideline this collaborative ethos often find themselves grappling with inefficiencies, even if they possess advanced tools. Fostering Collaboration: Just as continuous integration and delivery are integral to DataOps, so is continuous feedback. Regularly collecting feedback from team members and stakeholders and acting on it refines the DataOps process over time. Feedback Loops: In the fast-evolving world of data, learning never stops. Promoting a culture where team members are encouraged to learn, experiment, and innovate ensures that the organization remains at the forefront of data management best practices. Lifelong Learning: In summary, while DataOps holds the promise of revolutionizing , the key to unlocking this potential lies in adhering to these best practices. They act as the bedrock upon which successful DataOps implementation is built, ensuring that data processes are agile, efficient, and in perfect harmony with business objectives. data engineering Conclusion The Transformative Potential of DataOps As we've journeyed through the intricacies of DataOps, one thing is clear: its transformative potential in the realm of data engineering is immense. This isn't just another buzzword or fleeting trend. It represents a paradigm shift. By incorporating DevOps principles into data workflows, organizations can experience heightened efficiency. Gone are the days of siloed operations, where data engineers, data scientists, and business stakeholders worked in isolation. DataOps bridges these divides, fostering a collaborative ecosystem where data-driven decisions are expedited, aligned with business objectives, and rooted in high-quality, reliable data. Future Outlook of DataOps Looking ahead, the future of DataOps seems promising and exciting. As technology continues to advance, we can anticipate the emergence of even more sophisticated tools that further simplify data engineering tasks, promoting automation and ensuring even tighter integration of data processes. Moreover, as organizations globally recognize the merits of DataOps, we might witness the development of new methodologies, best practices, and standards that further refine this discipline. Furthermore, the growing emphasis on and machine learning will likely intertwine with DataOps. This union will give birth to intelligent data operations, where predictive analytics, automation, and adaptive data workflows become the norm. artificial intelligence In conclusion, DataOps stands at the cusp of revolutionizing the world of data engineering. Its principles, methodologies, and practices hold the key to navigating the complexities of today's data-driven world. As organizations continue to embrace and evolve with DataOps, the future of data engineering looks bright, collaborative, and extraordinarily efficient.

DataOps: the Future of Data Engineering

ডেটাঅপস: ডেটা ইঞ্জিনিয়ারিংয়ের ভবিষ্যত

DataOps: die Zukunft des Data Engineering

DataOps: el futuro de la ingeniería de datos

DataOps : l'avenir de l'ingénierie des données

डेटाऑप्स: डेटा इंजीनियरिंग का भविष्य

DataOps: データ エンジニアリングの未来

DataOps: 데이터 엔지니어링의 미래

DataOps: o futuro da engenharia de dados

DataOps: будущее инженерии данных

DataOps: Veri Mühendisliğinin Geleceği

DataOps: Tương lai của Kỹ thuật dữ liệu

DataOps：数据工程的未来

Introduction: Data engineering has evolved into specialized fields like DataOps and VectorOps to address the challenges of managing data and vector embeddings, respectively. As large language models (LLMs) like GPT-4 emerge, managing high-dimensional data structures becomes increasingly important. LangChain, an LLM-powered application development framework, integrates with DataOps and VectorOps processes and utilizes vector databases to create data-aware, interactive applications. DataOps: Streamlining Data Management and Processes: DataOps focuses on enhancing collaboration, integration, and automation of data flows across organizations. Implementing DataOps practices promotes data accuracy, consistency, and accessibility, which are vital for data-aware applications using LLMs. Key DataOps technologies include Apache Kafka for real-time data streaming, Apache Airflow for workflow management, and Apache NiFi for data routing and transformation. References: https://medium.com/codex/5-steps-to-implementing-a-successful-dataops-practice-f553effd7367 https://www.forbes.com/sites/forbesbusinessdevelopmentcouncil/2022/01/24/how-to-implement-a-successful-dataops-strategy-and-why-it-matters/?sh=56e354d5e87c https://www.linkedin.com/pulse/dataops-practical-understanding-implementation-taaha-khan-/ VectorOps: Tackling High-Dimensional Data Challenges: VectorOps specializes in managing vector embeddings, a critical component of LLMs. Traditional databases struggle with high-dimensional data structures, whereas vector databases excel in handling them. Integrating VectorOps with DataOps processes enables efficient large-scale vector embedding management, paving the way for innovative applications using LLMs. Popular vector database technologies used in VectorOps include FAISS, Milvus, and Pinecone. References: https://thenewstack.io/vector-databases-long-term-memory-for-artificial-intelligence https://www.pinecone.io/learn/vector-database/ https://techgenix.com/rise-of-vector-databases/ LangChain: Building Data-Aware LLM Applications: LangChain facilitates the development of applications that connect to data sources, interact with their environment, and utilize the power of LLMs. It offers various modules, such as Models, Prompts, Memory, Indexes, Chains, and Agents. The Indexes module focuses on structuring documents for optimal LLM interaction and retrieval, making it an excellent candidate for VectorOps and vector database integration. LangChain supports integration with LLMs like GPT-4, BERT, and RoBERTa. """
Example of creating a chat bot with GPT-Index & LangChain.
"""

from llama_index import SimpleDirectoryReader, GPTListIndex, readers, GPTSimpleVectorIndex, LLMPredictor, PromptHelper
from langchain import OpenAI
import sys
import os
from IPython.display import Markdown, display


def construct_index(directory_path):
    # set maximum input size
    max_input_size = 4096
    # set number of output tokens
    num_outputs = 2000
    # set maximum chunk overlap
    max_chunk_overlap = 20
    # set chunk size limit
    chunk_size_limit = 600 

    # define LLM
    llm_predictor = LLMPredictor(llm=OpenAI(temperature=0.5, model_name="text-davinci-003", max_tokens=num_outputs))
    prompt_helper = PromptHelper(max_input_size, num_outputs, max_chunk_overlap, chunk_size_limit=chunk_size_limit)
 
    documents = SimpleDirectoryReader(directory_path).load_data()
    
    index = GPTSimpleVectorIndex(
        documents, llm_predictor=llm_predictor, prompt_helper=prompt_helper
    )

    index.save_to_disk('index.json')

    return index

def ask_ai():
    index = GPTSimpleVectorIndex.load_from_disk('index.json')
    while True: 
        query = input("What do you want to ask? ")
        response = index.query(query, response_mode="compact")
        display(Markdown(f"Response: <b>{response.response}</b>")) References: https://python.langchain.com/en/latest/index.html https://github.com/hwchase17/langchain https://github.com/gkamradt/langchain-tutorials/blob/main/LangChain Cookbook Part 2 - Use Cases.ipynb Integrating DataOps, VectorOps, and LangChain: DataOps, VectorOps, and LangChain integration create powerful applications that combine efficient data management, high-dimensional data processing, and LLM capabilities. Some key advantages of this integration include: Enhanced similarity search: LangChain's integration with vector databases and VectorOps enables efficient near-neighbor matching, providing comprehensive results beyond traditional keyword-based searches. Scalability: DataOps and VectorOps processes ensure the efficient handling of large-scale vector embeddings, making the framework ideal for managing data in the era of LLMs and other complex models. """
As described here: https://python.langchain.com/en/latest/modules/indexes/vectorstores/examples/pinecone.html
"""

import pinecone
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.text_splitter import CharacterTextSplitter
from langchain.vectorstores import Pinecone
from langchain.document_loaders import TextLoader

loader = TextLoader('../../../state_of_the_union.txt')
documents = loader.load()
text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
docs = text_splitter.split_documents(documents)

embeddings = OpenAIEmbeddings()

# initialize pinecone
pinecone.init(
    api_key=PINECONE_API_KEY,  # find at app.pinecone.io
    environment=PINECONE_ENV  # next to api key in console
)

index_name = "langchain-demo"

docsearch = Pinecone.from_documents(docs, embeddings, index_name=index_name)

# if you already have an index, you can load it like this
# docsearch = Pinecone.from_existing_index(index_name, embeddings)

query = "What did the president say about Ketanji Brown Jackson"
docs = docsearch.similarity_search(query)

print(docs[0].page_content) Flexibility: LangChain allows users to perform searches based on high-level descriptions rather than relying on specific keywords or metadata classifications, enabling more versatile applications. Reference: https://mattboegner.com/knowledge-retrieval-architecture-for-llms/ https://simonwillison.net/2023/Jan/13/semantic-search-answers/ https://www.codingthesmartway.com/unlocking-the-power-of-llms-with-langchain-a-beginners-guide/ https://python.langchain.com/en/latest/modules/models/llms/integrations.html Conclusion: By focusing on DataOps, VectorOps, and LangChain, developers can create advanced, data-driven applications that leverage the power of LLMs and vector embeddings. Efficient data management through DataOps, coupled with VectorOps' expertise in high-dimensional data processing, unlocks the full potential of LLMs like GPT-4, BERT, and RoBERTa. LangChain serves as the bridge between these two fields, enabling developers to build innovative applications that can revolutionize industries and reshape the future of data-driven solutions. https://www.youtube.com/watch?v=UO699Szp82M&embedable=true As the need for data-aware applications and large-scale vector embedding management grows, the synergistic integration of DataOps, VectorOps, and LangChain will play a critical role in shaping the next generation of data-driven applications. By keeping up-to-date with the latest developments in these fields and embracing cutting-edge technologies, developers can stay ahead of the curve and lead the way toward a data-centric future.

The New Data Engineering Landscape: DataOps, VectorOps, and LangChain

Some 95 percent of businesses say they struggle with processing complex data, such as unstructured records and multi-source flows, according to . At the same time, about 90 percent of all data out . This is in itself indicative of the data conundrum facing businesses today. Data is growing more complex by the day, and to keep up with it, companies are forced to build, operate, and constantly upgrade a sophisticated low-latency pipeline infrastructure. TechJury there is unstructured Building even a simple Extract, Transform, Load (ETL) pipeline requires tireless hours of coding—several weeks on average. Furthermore, ETL requires efforts from multiple teams. First, data engineers must build data collection tools and ensure data validity. Software engineers must then develop the pipelines to pull the data into warehouses or lakes for analysts, the end-users, to make sure it is aligned with the business question. If it isn’t, as it happens way too often, the process goes back to square one, bleeding the company’s budget dry. The resulting workflow is linear and rigid: developers are usually forced to wait before data engineers are done, and analysts, for their part, have to wait for them both. Analysts cannot generate insights without enough data, which in turn requires a pipeline up and running. So what ends up happening is too many teams have nothing to do as the expenses continue to tick up. Even worse, in more dynamic environments, data collected with so much effort and investment can simply become outdated before the set is even complete. Furthermore, in my own experience, even the most business-minded tech enthusiasts are seldom excited about data. Developers would much rather focus on the product’s functionality than the boring data collection itself. Data engineers prefer handling data integrity, their core task, to the company’s informational plumbing. And as for data scientists, I imagine few of them get their fix spending . hours loading and cleaning the data More than just lost time The flaws of this approach stretch beyond the realms of wasted time and high costs. Today’s data-management philosophy is code-centric, meaning everyone involved, including the analysts, must be a coder. Data and software engineers write code to start capturing and preprocessing data. Then, analysts and data scientists must write code to fetch it and make the extra transformations needed for analysis or training models. Even in the rare case where all of them are outstanding coders, the result is still sluggish due to the use of Python and Java, interpreted languages that are generally slower than compiled ones. Another implication is that the legacy companies trying to tap data, the data immigrants, face an increasingly uphill struggle. They often lack the tech aptitude of their new-age counterparts, and the colossal price tag and time associated with developing data infrastructure will only weigh them down more. By the same account, companies who want to tap new data sources have to go through the full process as well, which slows down their progress. Tougher still, nobody is really able to innovate with data simply because even the most basic analysis requires substantial investment. Innovation takes time and requires all hands on deck, but with today’s DataOps workflow, both are a luxury. Day-to-day tasks simply take all of that away, leaving data teams no room for experimentation and more creative ideas. The problem comes down to the reliance on the ETL mantra. Building a rigid data infrastructure made up of pipelines tailored to very specific business questions with limited transformation abilities only perpetuates the conundrum at hand. It is heavy on resources and hampers productivity and innovation while failing to drive real progress. To truly unlock their potential, companies looking to tap data must embrace a whole new design and management paradigm. The simplistic ETL just doesn’t cut it. Businesses need data playgrounds, solutions that will make the necessary processed data instantly accessible to those in the enterprise who need it. Building the data playground At its core, a data playground is a strategy that moves the focus from code to the data itself, making it easily accessible to relevant teams with little need for further development. It is rooted in the idea of collaborative design that fosters cooperation between teams instead of locking them into a rigid linear sequence of tasks. While all teams retain their overall functions, the process breaks up the dependencies between them, allowing all of them to focus on their own tasks. At the core of the strategy is the playground itself, a software platform powering all of the company’s data operations. Software engineers will use this platform to develop Data Apps, which would feature both extraction and transformation capabilities. Data engineers would then use these Apps to produce validated data sources, ready to fuel the analysis by data scientists, who will be able to make extra transformations using the App’s built-in functions, if needed. The approach leaves the code-heavy tasks on the backend to the engineers, while the analysts are free to use the low-code to no-code front-end to build up new pipelines and transform data on the go. The playground brings more than the basic ETL to the table since it also allows the analysts to configure, redesign, and create a data flow on the go, within hours, not weeks. By collecting the backlog of Data Apps and data sources, the company can conduct integrated analysis to streamline the data operations and workflows, enriching the playground in an offline manner. Businesses looking to scale up with data need to understand success is not just about collecting as much data as possible. It is just as much, if not even more, about how you manage what you have collected, and whether you can make changes and amendments on the go without having to rebuild the entire pipeline and waiting for more data. Granted, it’s a careful balancing act. But to be flexible, innovative, and ultimately effective, businesses must manage their DataOps in a way that plays to the employees’ strengths and makes data easy for them to access, not gathering it for months and years on end with no use.

Data Playgrounds are The Cure for Slow and Inefficient DataOps

Procesando Datos Para Deep Learning: Datasets, visualizaciones y DataLoaders en PyTorch

Por favor clic el artículo original： http://www.octoparse.es/blog/70-fuentes-de-datos-gratuitas-en-2020 Todos gran visualización de datos comienza con datos buenos y limpios. La mayoría de la gente cree que recopilar grandes datos sería un trabajo difícil, pero simplemente no es cierto. Hay miles de conjuntos de datos gratuitos disponibles en línea website, listos para ser analizados y visualizados por cualquier persona. Aquí hemos reunido 70 fuentes de datos gratuitas para 2020 sobre , gobierno, delincuencia, salud, datos financieros y económicos, marketing y redes sociales, periodismo y medios, bienes raíces, directorio, revisión de empresas y más. Fuente de Datos Gratuita: Gobierno : Es la primera etapa y proporciona información gratuita sobre el clima y la delincuencia en la web por parte del gobierno de los Estados Unidos. Data.gov : Aquí hay conjuntos de datos de todos los departamentos centrales del Reino Unido, así como de muchas otras autoridades locales y públicas. Actúa como un portal a todo tipo de información sobre todo, incluyendo negocios y economía, crimen y justicia, defensa, educación, medio ambiente, gobierno, salud, sociedad y transporte. Data.gov.uk : Este sitio contiene las últimas estadísticas gubernamentales sobre la vida de los ciudadanos estadounidenses, incluida la población, la economía, la educación, la geografía y más. U.S. Census Bureau : Datos sobre todos los países del mundo; se enfoca en temas de historia, gobierno, población, economía, energía, geografía, comunicaciones, transporte, militares y transnacionales para 267 países. The CIA World Factbook : Socrata es una compañía de software impulsada por la misión que es otro lugar interesante para explorar datos relacionados con el gobierno con algunas herramientas de visualización integradas. Sus datos como servicio han sido adoptados por más de 1200 agencias gubernamentales para datos abiertos, gestión del desempeño y gobierno basado en datos. Socrata : Portal de datos abiertos de la Unión Europea: es el único punto de acceso a una creciente gama de datos de las instituciones y otros organismos de la Unión Europea. Los aumentos de datos incluyen el desarrollo económico dentro de la UE y la transparencia dentro de las instituciones de la UE, incluidos datos geográficos, geopolíticos y financieros, estadísticas, resultados electorales, actos jurídicos y datos sobre delincuencia, salud, medio ambiente, transporte e investigación científica. Podrían reutilizarse en diferentes bases de datos e informes. Y más, una variedad de formatos digitales están disponibles en las instituciones de la UE y otros organismos de la UE. El portal proporciona un catálogo estandarizado, una lista de aplicaciones y herramientas web que reutilizan estos datos, un editor de consultas de punto final SPARQL y acceso a la rest API, y consejos sobre cómo hacer el mejor uso del sitio. European Union Open Data Portal es un proyecto piloto con muchos conjuntos de datos gubernamentales y geoespaciales. Le ayuda a explorar cómo el gobierno de Canadá crea una mayor transparencia, responsabilidad, aumenta la participación ciudadana e impulsa la innovación y las oportunidades económicas a través de datos abiertos, información abierta y diálogo abierto. Canada Open Data : ofrece datos abiertos del gobierno de US, La UE, Canadá, CKAN y más. Datacatalogs.org : El Centro Nacional de Estadísticas de Educación (NCES) es la principal entidad federal para recopilar y analizar datos relacionados con la educación en los U.S./otras naciones. U.S. National Center for Education Statistics : La colección del Servicio de datos del Reino Unido incluye importantes encuestas patrocinadas por el gobierno del Reino Unido, encuestas transnacionales, estudios longitudinales, datos del censo del Reino Unido, agregado internacional, datos comerciales y datos cualitativos. UK Data Service Fuente de Datos Gratuita: Crimen : El Programa UCR ha sido el punto de partida para ejecutivos de la ley, estudiantes, investigadores, miembros de los medios de comunicación y el público que busca información sobre delitos en los Estados Unidos. Uniform Crime Reporting : Statistical crime informes y publicaciones estadísticas sobre delitos que detallan delitos específicos y describen tendencias para comprender las amenazas de delitos a nivel local y nacional. FBI Crime Statistics : información sobre todo lo relacionado con el sistema de justicia penal de los U.S., Incluidas las muertes relacionadas con el arresto, el censo de los reclusos en la cárcel, la encuesta nacional de laboratorios de delitos de ADN, Investigaciones de organismos encargados de hacer cumplir la ley, etc. Bureau of Justice Statistics : Búsqueda nacional de delincuentes sexuales: es un recurso de seguridad pública sin precedentes que permite el acceso público a los datos de delincuentes sexuales en todo el país. Presenta la información más actualizada proporcionada por cada Jurisdicción. National Sex Offender Search Fuente de Datos Gratuita: Salud : Aquí encontrará un archivo de datos comprimido de la base de datos Drugs@FDA. Drugs@FDA se actualiza diariamente, y este archivo de datos se actualiza una vez por semana, el martes. U.S. Food & Drug Administration : UNICEF gathers evidence on the situation of children and women around the world. The data sets include accurate, nationally representative data from household surveys and other sources. UNICEF : Estadísticas sobre nutrición, enfermedades y salud en más de 150 países. : 125 años de datos de atención médica de EE. UU., Incluidos datos de Medicare a nivel de reclamos, epidemiología y estadísticas de población. World Health Organisation Healthdata.gov : Conjuntos de datos de salud del Servicio Nacional de Salud del Reino Unido. La organización produce más de 260 publicaciones estadísticas oficiales y nacionales. Esto incluye datos comparativos nacionales para usos secundarios, desarrollados a partir de las estadísticas de episodios long-running hospitalarios que pueden ayudar a los tomadores de decisiones locales a mejorar la calidad y la eficiencia de la atención de primera línea. NHS Health and Social Care Information Centre Fuente de Datos Gratuita: datos financieros y económicos : Estadísticas educativas sobre todo, desde finanzas hasta indicadores de prestación de servicios. World Bank Open Data :Una fuente de información increíblemente útil que incluye informes de estabilidad financiera mundial, informes económicos regionales, estadísticas financieras internacionales, tipos de cambio, dirección comercial y más. IMF Economic Data : Acceso gratuito a datos detallados del comercio mundial con visualizaciones. UN Comtrade es un depósito de estadísticas oficiales de comercio internacional y tablas analíticas relevantes. Se puede acceder a todos los datos a través de la API. UN Comtrade Database : Con datos sobre más de 60,000 compañías que cubren 300 años, Global Financial Data proporciona una fuente especial única para analizar los giros y vueltas de la economía global. Global Financial Data : Cotizaciones y gráficos de acciones en tiempo real, noticias financieras, conversiones de divisas o carteras rastreadas Google Finance . : El Explorador de datos públicos de Google proporciona datos públicos y pronósticos de una variedad de organizaciones internacionales e instituciones académicas, incluidos el Banco Mundial, la OCDE, Eurostat y la Universidad de Denver. Estos se pueden mostrar como gráficos de líneas, gráficos de barras, diagramas de sección transversal o en mapas. Google Public Data Explorer :Estadísticas macroeconómicas y de la industria oficiales de U.S., Principalmente informes sobre el producto interno bruto (PIB) de los Estados Unidos y sus diversas unidades. También proporcionan información sobre ingresos personales, ganancias corporativas y gastos del gobierno en sus Cuentas Nacionales de Ingresos y Productos (NIPAs). U.S. Bureau of Economic Analysis :Plentiful links to anything related to finance, no matter how obscure, including World Development Indicators Online, World Bank Open Data, Global Financial Data, International Monetary Fund Statistical Databases, y . Financial Data Finder at OSU EMIS Intelligence : Datos macro, datos de la industria, datos de productividad, datos comerciales, finanzas internacionales, datos y más. National Bureau of Economic Research : Conjuntos de datos trimestrales de información extraída de las exhibiciones a informes financieros corporativos presentados a la Comisión. : Visualizaciones de datos sobre la economía. U.S. Securities and Exchange Commission Visualizing Economics :Financial Times proporciona una amplia gama de información, noticias y servicios para la comunidad empresarial global. Financial Times Fuente de Datos Gratuita: Marketing y Redes Sociales : Explore los conjuntos de datos públicos de Amazon Web Services por categoría para obtener una gran cantidad de información. Amazon API Gateway permite a los desarrolladores conectar de forma segura aplicaciones móviles y web a API que se ejecutan en Amazon Web (AWS) Lambda, Amazon EC2 u otros servicios web de acceso público que se alojan fuera de AWS. Amazon API : ASTA es la asociación de profesionales de viajes más grande del mundo. Proporciona información a los miembros, incluidas las agencias de viajes y las empresas cuyos productos venden, como tours, cruceros, hoteles, alquiler de automóviles, etc. American Society of Travel Agents :Social Mention es una plataforma de búsqueda y análisis de redes sociales que agrega contenido generado por los usuarios de todo el universo en un solo flujo de información. Social Mention : Google Trends muestra la frecuencia de ingresar términos de búsqueda específicos en varios idiomas en relación con el volumen de búsqueda total en todo el mundo. Google Trends : Aprenda a publicar y recuperar datos de Facebook utilizando la API Graph. Facebook API : La plataforma de Twitter conecta su sitio web o aplicación con la conversación mundial que tiene lugar en Twitter. : La plataforma de API de Instagram se puede utilizar para crear aplicaciones y servicios sin automatizados, auténticos y de alta calidad. Twitter API Instagram API : La API de Foursquare le brinda acceso a nuestra base de datos de lugares de clase mundial y la capacidad de interactuar con los usuarios y comerciantes de Foursquare. Foursquare API : Un gran depósito de datos de marketing. Puede encontrar las últimas estadísticas y tendencias de marketing aquí. También proporciona herramientas para marketing en redes sociales, gestión de contenido, análisis web, páginas de destino y optimización de motores de búsqueda. HubSpot : Información sobre SEO que incluye investigación de palabras clave, creación de enlaces, auditorías de sitios e información de optimización de páginas para ayudar a las empresas a tener una mejor visión de la posición que tienen en los motores de búsqueda y cómo mejorar su clasificación. Moz : Las últimas noticias, estudios e investigaciones sobre marketing de contenidos. Content Marketing Institute Fuente de Datos Gratuita: Periodismo y Medios – Busque artículos de Times desde 1851 hasta hoy, recuperando titulares, resúmenes y enlaces a multimedia asociados. También puede buscar reseñas de libros, listados de eventos de Nueva York, críticas de películas, historias principales con imágenes y más. The New York Times Developer Network :La de contenido AP le permite buscar y descargar contenido utilizando sus propias herramientas editoriales, sin tener que visitar los portales AP. Proporciona acceso a imágenes de AP, propiedad de un miembro y de terceros, y videos producidos por AP y terceros seleccionados. Associated Press API API : Es un motor de búsqueda en línea que registra las frecuencias de cualquier conjunto de cadenas de búsqueda delimitadas por comas utilizando un recuento anual de n-gramos encontrados en fuentes impresas entre 1500 y 2008 en los corpus de texto de Google. Google Books Ngram Viewer : Wikipedia offers free copies of all available content to interested users. Wikipedia Database : Es un sitio web que se centra en el análisis de encuestas de opinión, política, economía y blogs de deportes. LLos datos y el código en Github están detrás de la historia y la interacción de FiveThirtyEight. FiveThirtyEight : Google Scholar es un motor de búsqueda web de acceso gratuito que indexa el texto completo o los metadatos de la literatura académica en una variedad de formatos y disciplinas de publicación. Incluye la mayoría de las revistas y libros académicos en línea revisados ​​por pares, documentos de conferencias, tesis y disertaciones, preimpresiones, resúmenes, informes técnicos y otra literatura académica, incluidas las opiniones de los tribunales y las patentes. Google Scholar Free Data Source: Real Estate : Castles son una exitosa agencia independiente de propiedad privada. Establecidos en 1981, ofrecen un servicio integral que incorpora ventas residenciales, alquileres y administración, y encuestas y valoraciones. Castles : RealEstate.com sirve como el mejor recurso para los compradores de vivienda por primera vez, ofreciendo herramientas fáciles de entender y asesoramiento experto en cada etapa del proceso. Realestate.com : Gumtree es el primer sitio de anuncios clasificados gratuitos en el Reino Unido. Comprar y vender artículos, automóviles, propiedades y encontrar u ofrecer trabajos en su área está disponible en el sitio web. Gumtree : Proporciona un enfoque innovador de base de datos para ventas residenciales, alquileres y administración. James Hayward : Japan’s property website. Lifull Home’s : Italy’s property website. Immobiliare.it : Italy’s property website. Subito : El sitio web inmobiliario líder de Bélgica. Immoweb Fuente de Datos Gratuita: Directorio de Negocios y Revisión : LinkedIn es un servicio de redes sociales orientado a los negocios y el empleo que opera a través de sitios web y aplicaciones móviles. Tiene 500 millones de miembros en 200 países y puede encontrar el directorio de negocios aquí. LinkedIn : OpenCorporates es la base de datos abierta más grande de compañías y datos de compañías en el mundo, con más de 100 millones de compañías en un número similar de jurisdicciones. Nuestro objetivo principal es hacer que la información sobre las empresas sea más utilizable y esté más ampliamente disponible para el beneficio público, particularmente para abordar el uso de las empresas con fines criminales o antisociales, por ejemplo, corrupción, lavado de dinero y crimen organizado. OpenCorporates : La fuente original para encontrar y conectarse con fontaneros locales, personal de mantenimiento, mecánicos, abogados, dentistas y más. Yellowpages : Craigslist es un sitio web estadounidense de anuncios clasificados con secciones dedicadas a empleos, vivienda, anuncios personales, en venta, artículos buscados, servicios, comunidad, conciertos, currículums y foros de discusión. Craigslist : Fundado en 1886, GAF se ha convertido en el mayor fabricante de techos comerciales y residenciales de América del Norte (Fuente: estudio del Grupo Fredonia). Nuestro éxito en el crecimiento de la compañía a casi $ 3 mil millones en ventas ha sido el resultado de nuestra incesante búsqueda de la calidad, combinada con la experiencia líder en la industria y soluciones integrales para techos. Jim Schnepper es el presidente de GAF, una subsidiaria operativa de Standard Industries. Cuando busca proteger las cosas que más valora, estas son solo algunas de las razones por las que creemos que debe elegir GAF. GAF Master Elite Contractor : Aquí puede encontrar contratistas, remodeladores, instaladores o constructores en los US y Canadá en su proyecto residencial o comercial. CertainTeed : Toda la información sobre las empresas en California. Companies in California : Manta es uno de los mayores recursos en línea que ofrece productos, servicios y educativas oportunidades. El directorio de Manta cuenta con millones de visitantes únicos cada mes que buscan en la base de datos integral empresas individuales, segmentos de la industria y listados geográficos específicos. Manta : Directorio sobre startups en la UE.、 EU-Startups : Directorio de abogados. La Asociación de Abogados de Kansas (KBA) se fundó en 1882 como una asociación voluntaria para profesionales legales dedicados y tiene más de 7,000 miembros, incluidos abogados, jueces, estudiantes de derecho y asistentes legales. Kansas Bar Association Fuente de datos gratuita: Otros sitios web del portal : Directorio sobre software de negocios y reseñas. Capterra : Fuente de datos para empleos y oportunidades de carrera. Monster : Directorio sobre trabajos e información sobre información privilegiada sobre empresas con revisiones de empleados, herramientas salariales personalizadas y más. Glassdoor : Directorio sobre servicio de automóviles, MOT o reparación de automóviles The Good Garage Scheme : Información sobre fragancias. OSMOZ : Una herramienta de extracción de datos gratuita para recopilar todos los datos web mencionados anteriormente en línea. Octoparse

Big Data: 70 Increíbles Fuentes de Datos Gratuitas que Debes Conocer para 2020

বিগ ডেটা: 70টি অবিশ্বাস্য ফ্রি ডেটা উত্স যা আপনার 2020 এর জন্য জানা উচিত

Big Data: 70 unglaubliche kostenlose Datenquellen, die Sie für 2020 kennen sollten

Big Data : 70 sources de données gratuites incroyables à connaître pour 2020

बड़ा डेटा: 70 अविश्वसनीय मुफ़्त डेटा स्रोत जो आपको 2020 के लिए जानना चाहिए

ビッグデータ: 2020 年に知っておくべき 70 の驚くべき無料データ ソース

빅 데이터: 2020년에 알아야 할 70가지 놀라운 무료 데이터 소스

Big Data: 70 fontes de dados gratuitas incríveis que você deve conhecer para 2020

Большие данные: 70 невероятных бесплатных источников данных, которые вы должны знать в 2020 году

Büyük Veri: 2020 İçin Bilmeniz Gereken 70 İnanılmaz Ücretsiz Veri Kaynağı

Dữ liệu lớn: 70 nguồn dữ liệu miễn phí đáng kinh ngạc bạn nên biết cho năm 2020

大数据：2020 年您应该了解的 70 个令人难以置信的免费数据源

La necesidad de crawling datos web ha aumentado en los últimos años. Los datos crawled se pueden usar para evaluación o predicción en diferentes campos. Aquí, me gustaría hablar sobre 3 métodos que podemos adoptar para scrape datos desde un sitio web. 1. Utilice Website APIs Muchos sitios web de redes sociales grandes, como Facebook, Twitter, Instagram, StackOverflow, proporcionan API para que los usuarios accedan a sus datos. A veces, puede elegir las API oficiales para obtener datos estructurados. Como se muestra a continuación en la Facebook Graph API, debe elegir los campos donde realiza la consulta, luego ordenar los datos, realizar la búsqueda de URL, realizar solicitudes, etc. Para obtener más información, puede consultar https://developers.facebook.com/docs/graph-api/using-graph-api. 2. Construye tu propio crawler Sin embargo, no todos los sitios web proporcionan API a los usuarios. Ciertos sitios web se niegan a proporcionar API públicas debido a límites técnicos u otras razones. Alguien puede proponer fuentes RSS, pero debido a que ponen un límite a su uso, no sugeriré ni haré más comentarios al respecto. En este caso, lo que quiero discutir es que podemos construir un crawler por nuestra cuenta para hacer frente a esta situación. ¿Cómo funciona un crawler? Un crawler, dicho de otra manera, es un método para generar una lista de URL que puede alimentar a través de su extractor. Los crawlers se pueden definir como herramientas para encontrar las URL. Primero le da al crawler una página web para que comience, y ellos seguirán todos estos enlaces en esa página. Entonces este proceso continuará en un bucle. Entonces, podemos proceder a construir nuestro propio crawler. Se sabe que Python es un lenguaje de programación de código abierto, y puede encontrar muchas bibliotecas funcionales útiles. Aquí, sugiero BeautifulSoup (Python Library) porque es más fácil de trabajar y posee muchos caracteres intuitivos. Más exactamente, utilizaré dos módulos de Python para crawl los datos. BeautifulSoup no obtenar la página web para nosotros. Por eso uso urllib2 para combinar con la biblioteca BeautifulSoup. Luego, debemos lidiar con las etiquetas HTML para encontrar todos los enlaces dentro de las etiquetas <a> de la página y la tabla derecha. Después de eso, repita cada tr (td)  y luego asigne cada elemento de tr (td) a una variable y añádalo a una lista. Primero veamos la estructura HTML de la tabla (no voy a extraer información para la título de la tabla <th>). Al adoptar este enfoque, su crawler se personaliza. Puede lidiar con ciertas dificultades encontradas en la extracción de API. Puede usar el proxy para evitar que algunos sitios web lo bloqueen, etc. Todo el proceso está bajo su control. Este método debería tener sentido para las personas con habilidades de codificación. El marco de datos que crawled debe ser como la figura a continuación. 3. Aproveche las herramientas de ready-to-use crawler Sin embargo, crawl un sitio web por su cuenta mediante la programación puede llevar mucho tiempo. Para las personas sin habilidades de codificación, esta sería una tarea difícil. Por lo tanto, me gustaría presentar algunas Crawler Tools. Octoparse Octoparse es un poderoso web data crawler basado en Windows. Es realmente fácil para los usuarios comprender esta herramienta con su interfaz de usuario simple y amigable. Para usarlo, debe descargar esta aplicación en su escritorio local. Como se muestra en la figura a continuación, puede hacer clic y arrastrar los bloques en el panel Workflow Designer para personalizar su propia tarea. Octoparse ofrece dos ediciones de planes de suscripción de crawling service: la Edición Gratuita y la Edición de Pago. Ambos pueden satisfacer las necesidades básicas de scraping o crawling de los usuarios. Con la edición gratuita, puede ejecutar sus tareas en el lado local. Si cambia su edición gratuita a una Edición de Pago, puede usar el cargando sus tareas en la Plataforma de la nube. De 6 a 14 servidores en la nube ejecutarán sus tareas simultáneamente con una mayor velocidad y crawl a mayor escala. Además, puede automatizar su extracción de datos sin dejar rastro utilizando la función de proxy anónimo de Octoparse que podría rotar toneladas de IP, lo que evitará que ciertos sitios web lo bloqueen. Aquí hay un video que presenta Octoparse Cloud Extraction. Cloud-based service Octoparse también proporciona para conectar su sistema a sus datos scraped en tiempo real. Puede importar los datos de Octoparse en su propia base de datos o usar la API para solicitar acceso a los datos de su cuenta. Después de finalizar la configuración de la tarea, puede exportar datos a varios formatos, como CSV, Excel, HTML, TXT y database (MySQL, SQL Server y Oracle). API Import.io Import.io también se conoce como un web crawler que cubre todos los diferentes niveles de necesidades de crawling. Ofrece una herramienta mágica que puede convertir un sitio en una formar sin ninguna sesión de entrenamiento. Sugiere a los usuarios descargar su aplicación de escritorio si es necesario crawl sitios web más complicados. Una vez que haya creado su API, ofrecen una serie de opciones de integración simples, como  Google Sheets, Plot.ly, Excel, así como solicitudes GET y POST. Cuando considera que todo esto viene con una etiqueta de precio de por vida y un equipo de soporte increíble, import.io es un primer elección claro para aquellos que buscan datos estructurados. También ofrecen una opción paga de nivel empresarial para empresas que buscan una extracción de datos más compleja o a gran escala. Mozenda Mozenda es otro extractor de datos web fácil de usar. Tiene una interfaz de usuario de apuntar y hacer clic para usuarios sin ninguna habilidad de codificación para usar. Mozenda también elimina la molestia de automatizar y publicar datos extraídos. Dígale a Mozenda qué datos desea una vez, y luego obténgalos con la frecuencia que los necesite. Además, permite una programación avanzada utilizando REST API que el usuario puede conectar directamente con la cuenta Mozenda. Proporciona el servicio basado en la nube y la rotación de IP también. ScrapeBox SEO expertos, los vendedores en línea e incluso los spammers deberían estar muy familiarizados con ScrapeBox con su interfaz de usuario muy fácil de usar. Los usuarios pueden recolectar fácilmente datos de un sitio web para recibir correos electrónicos, verificar el rango de la página, verificar los servidores proxy y el envío de RSS. Mediante el uso de miles de servidores proxy rotativos, podrá escabullirse de las palabras clave del sitio de la competencia, investigar en sitios .gov, recopilar datos y comentar sin ser bloqueado o detectado. Google Web Scraper Plugin Si la gente solo quiere scrape datos de una manera simple, le sugiero que elija el complemento Google Web Scraper. Es un web scraper basado en navegador que funciona como Outwit Hub de Firefox. Puede descargarlo como una extensión e instalarlo en su navegador. Debe resaltar los campos de datos que desea scrape, hacer clic con el botón derecho y elegir "Scrape similar ...". Todo lo que sea similar a lo que destacó se representará en una tabla lista para exportar, compatible con Google Docs. La última versión todavía tenía algunos errores en las hojas de cálculo. Aunque es fácil de manejar, no puede scrape imágenes y crawl datos en gran cantidad.

3 Mejores Formas de Crawl Datos desde Website

Para extraer datos de websites, puede usar las herramientas de extracción de datos como . Estas herramientas pueden extraer datos de website automáticamente y guardarlos en muchos formatos, como Excel, JSON, CSV, HTML o en su propia base de datos a través de . Solo toma unos minutos puede extraer miles de líneas de datos, la mejor es que no se necesita codificación en este proceso. Octoparse API Tabla de contenidos Método #1: Extracción Sencilla con Las Plantillas de Web Scraping Método #2: Extracción Personalizada con Modo Avanzado Tome Google Search como ejemplo. Si estamos interesados en información sobre el "smoothie" y queremos extraer todos los títulos, descripciones y URLs de los resultados de búsqueda. Para extraer datos de Google Search, puede usar una . plantilla de web scraping La plantilla es un crawler preformateado que listo para usar sin ninguna configuración. Hay más de 50 plantillas para elegir. Verá todas las plantillas desde sitios web de comercio electrónico como Amazon y eBay hasta canales de redes sociales como Facebook, Twitter e Instagram. Octoparse también ofrece plantillas personalizadas. Método #1: Extracción Sencilla con Las Plantillas de Web Scraping Primero: Elija una plantilla de web scraping Para usar la plantilla, necesita instalar Octoparse en la computadora. Seleccione el modo "Task Template". Vaya a la plantilla de web scraping de Google Search en la categoría "search engine", Segundo: lee las instrucciones de la plantilla Abre la plantilla. Consulte las instrucciones y la salida de muestra para asegurarse de que esta plantilla le proporcionará los datos que necesita. Puede pasar el cursor sobre los campos de datos para ver qué elementos de los sitios web se extraerán. Consulte los parámetros para comprender mejor lo que necesita ingresar. Los parámetros variarían en diferentes plantillas, porque necesitan diferentes términos de búsqueda para continuar. Puede ser una URL, una palabra clave, una lista de URL/palabras clave , la cantidad de páginas que desea obtener, etc. En este caso, debemos ingresar el término de búsqueda "smoothie" Tercero: Usa la plantilla y comienza datos extracción Sigue haciendo clic en "use template", luego ingrese "smoothie" y presione "save and run". Si es una-vez proyecto, simplemente puede ejecutar el crawler en su computadora local. Mientras que, si está manejando un proyecto en curso, puede extraer datos en la  plataforma de la nube de Octoparse. Después de la extracción, puede exportarla a muchos formatos, como Excel, CSV y txt. Acabamos de presentar cómo usar una plantilla de web scraping para extraer web datos de Google Search. También puede crear su propio crawler utilizando el "Modo avanzado". Es posible que necesita algunas configuraciones, pero es muy flexible en términos de extracción de datos. Método #2: Extracción Personalizada con Modo Avanzado Paso 1: Ingrese la URL de destino para crear un crawler Si está tratando de , puede ingresar una lista de hasta 10,000 URL en el cuadro. En este caso, como solo estamos scraping solo un sitio web, peguemos la URL de destino en el cuadro y haga clic en "save URL" para continuar. extraer datos a gran escala Paso 2: Crea un bucle de paginación Cambie el navegador a Firefox 45. Ahora Octoparse ha cargado la página web en el navegador incorporado con éxito. Luego, debemos crear una paginación haciendo clic en el botón de página "Next" y seleccionando "Loop click next page" en el panel de Consejos de acción. Verá el bucle de paginación que acabamos de crear en el área de flujo de trabajo. Paso 3: Extraer los datos e iniciar la extracción Ahora podemos extraer los datos. Haga clic en el título de un resultado de búsqueda y haga clic en "select all". Después de seleccionar todos los títulos, se resaltarán en verde. Haga clic en "extract text of the selected element" para extraer todos los títulos. Hagamos una pausa por un momento para ver el flujo de trabajo. Como veis, acabamos de construir un bucle extracción dentro del ciclo de paginación. Todo el proceso de extracción funcionará de esta manera: el bot primero abrirá la página web, extraerá los títulos en la primera página uno por uno, y luego pasará a la página siguiente para repetir la extracción hasta que la extracción se detenga o se complete. Además de Google, las herramientas de extracción de datos pueden extraer datos de muchos otros sitios web, y se utilizan ampliamente en todas las industrias. Por ejemplo, las compañías pueden extraer mapas de Yellowpages, Yelp y Google para generar oportunidades de ventas. Puede consultar otros usos y aplicaciones de extracción de datos . Previously published at http://www.octoparse.es/blog/scrape-websites-sin-ser-bloqueado

Extraer Datos del Website a Excel Automáticamente

¿Alguna vez te sucede cuando la gente te pide que escribas una API separada para integrar datos de redes sociales y guardar los datos sin procesar en tu base de datos de análisis en el sitio? Definitivamente quieres saber qué es la , cómo se usa en y qué puede lograr con ella. Echemos un vistazo. API web scraping Tabla de contenidos ¿Qué es una API? API estándar y API avanzada ¿Qué es una API? Wikipedia dice que: “En la , una interfaz de programación de aplicaciones (API) es un conjunto de definiciones de , y herramientas para construir . En términos generales, es un conjunto de métodos de comunicación claramente definidos entre varios componentes de software" computer programming subrutinas protocols software de aplicaciones En general, la API web es un conjunto de reglas que los desarrolladores deben seguir cuando interactúan con un lenguaje de programación. Al igual que Harry Potter debe decir "Alohomora" para abrir una puerta. No es completamente cierto ya que solo es responsable de buscar los datos de acuerdo con los recursos dedicados. En la mayoría de los casos, obtendrá solo lo que solicita. Sin embargo, no tiene acceso a otra información Una idea errónea que la mayoría de la gente tiene es que API puede extraer datos. Web Scraping API Por ejemplo, si desea realizar un análisis de opinión y necesita revisiones y comentarios, se utiliza para enviar su solicitud de esa palabra clave a un servidor web y, a cambio, el servidor le proporciona revisiones o comentarios en un formato de datos sin procesar. Los datos de formato sin formato no necesariamente parecen fáciles de usar, como las filas y columnas de la hoja de cálculo. una API web Datos JSON sin procesar en cromo Como tal, para "consumir los datos" de una página de producto, debemos seguir algunos pasos para un proceso intacto de extracción, transformación al almacenamiento. A veces, incluso tiene que convertir los datos sin formato al formato deseado. Parece una tarea fácil para programadores experimentados. Sin embargo, la complejidad todavía frustra a las personas que no tienen experiencia en programación pero que necesitan más datos. API estándar y API avanzada Para reducir la complejidad, es mejor tener una herramienta de raspado web con alguna integración de API que pueda extraer y transformar los datos al mismo tiempo sin escribir ningún código. Octoparse es una herramienta intuitiva de web scraping diseñada para que los no codificadores extraigan datos de cualquier sitio web. Sus ingenieros de software crean la integración API para que pueda lograr dos cosas: 1. Extraiga los datos del sitio web sin la necesidad de esperar la respuesta de un servidor web. 2. Envíe los datos extraídos automáticamente de la nube a sus aplicaciones internas a través de la integración de API de Octoparse Además de la flexibilidad, le permite convertir datos sin procesar en formularios como Excel, CSV según lo necesite. Otro beneficio es que puede funcionar según lo planeado que elimina la complicidad durante la extracción manual de datos. En caso de que nunca haya usado Octoparse, permítame explicarle en detalle cómo puede usar Octoparse para extraer datos y transmitirlos a su base de datos. Octoparse tiene dos tipos de API. El primero es la . puede hacer todos los trabajos como mencioné anteriormente. Puede usarlo para extraer datos en un sistema CRM o una herramienta de visualización de datos para generar informes hermosos Standard API Una API estándar La segunda API se llama . Es un superconjunto de la API estándar. Hace todo lo que hace la API estándar. Mejor aún, puede acceder y manipular los datos almacenados en la nube. A medida que el modelo de negocio basado en datos se ha vuelto más popular, se espera que las personas sin conocimientos de codificación usen diferentes herramientas para extraer datos. Si también está frustrado al usar una API, encontrará un gran valor en ya que su proceso de integración es fácil Advanced API Octoparse

Web Scraping API para Extracción de Datos: Una Guía para Principiantes

The resurgence of SQL-based RDBMS If there is one analytics trend we can all agree on, it is that we’ve seen a vast increase in the amount of business data collected in the past decade. A typical business today uses dozens if not hundreds of cloud-based, subscription-as-a-service software, for all sorts of purposes. Together with more traditional in-house enterprise software, the average business produces a mountain of data today, compared to just a few years ago. The savvier businesses are the ones who have found great value in analyzing their data, in order to make informed decisions, causing a tremendous uptick in both the demand and the supply of data management systems. Up till a few years ago, the traditional way of managing data (in SQL-based RDBMSs) was considered a relic of the past, as these systems couldn’t scale to cope with such a huge amount of data. Instead, NoSQL systems like HBase, Cassandra, and MongoDB became fashionable as they marketed themselves as being scalable and easier to use. Today, there seems to be a resurgence of SQL-based database systems Yet today, there seems to be a resurgence of SQL-based database systems. According to Amazon’s CTO, its PostgreSQL and MySQL-compatible Aurora database product has been the “fastest-growing service in the history of AWS”. SQL-based MPP (massively parallel processing) data warehouses, which used to be expensive and could only be purchased by big enterprises, have now moved to the cloud and are increasingly commoditized. All the major cloud providers today have their own MPP data warehouses: Google has BigQuery, Amazon has Redshift and Microsoft has Azure Data Warehouse … and that’s not to mention Snowflake — an independent data warehouse vendor that was recently valued at $3.9 billion. Everyone can now spin up their own data warehouse … and pay by the minute, which was unheard of just a decade ago. These developments have brought tremendous power and unprecedented access to data analysts, for they may now execute complex analytical queries and get the results in seconds or minutes instead of hours like with traditional data warehouses. SQL for data analytics For data analytics, the reasons to this resurgence are simple to understand: NoSQL sucks at analytical queries: the lack of joins means it is almost impossible to write complex analytical queries on these databases. No standardized query language: each vendor creates their own language to query their NoSQL database, which adds friction and learning curves to analysts. SQL, on the other hand, is ubiquitous and standardized. Almost every data analyst is familiar with it. In Google’s own words about their massively scalable in-house data management system, Google Spanner: The original API of Spanner provided NoSQL methods for point lookups and range scans of individual and interleaved tables. While NoSQL methods provided a simple path to launching Spanner, and continue to be useful in simple retrieval scenarios, SQL has provided significant additional value in expressing more complex data access patterns and pushing computation to the data. Spanner’s SQL engine shares a common SQL dialect, called “Standard SQL”, with several other systems at Google including internal systems such as F1 and Dremel (among others), and external systems such as BigQuery… For users within Google, this lowers the barrier of working across the systems. A developer or data analyst who writes SQL against a Spanner database can transfer their understanding of the language to Dremel without concern over subtle differences in syntax, NULL handling, etc. Together with the rise of cloud-based MPP data warehouses, SQL has now become a very attractive way to do data analytics. Capitalizing on this trend, startups like Mode Analytics, Periscope Data and open source tools like Redash became more popular. For analysts which are proficient with SQL, they can now utilize the power of their brand new cloud-based data warehouses to produce beautiful charts and dashboards without the need to learn any proprietary language or tool. Furthermore, SQL code is just text and thus can be stored in a version control system, making it trivial to manage. The problems with SQL However, using SQL for data analytics is not all sunshine and roses. Any tool that relies heavily on SQL is off-limits to business users who want more than viewing static charts and dashboards but don’t know the language. It forces business users to fall back to old-school spreadsheets, where they export raw data to Excel and perform manual calculations on their own, causing problems around accuracy and data consistency down the line. SQL is also too low level, and queries quickly become convoluted when more complex analysis is required. Most SQL queries read just fine when the analysis requires a couple of tables. But when the number of related tables increases, data analysts are forced to keep track of multiple types of joins and choose the right one for the right situation every time they write a query. This implies that SQL is not reusable, causing similar code with slightly different logic to be repeated all over the place. For example, one cannot easily write a SQL ‘library’ for accounting purposes and distribute it to the rest of your team for reuse whenever accounting-related analysis is required. Traditional data modeling The traditional way of providing BI to non-technical users is to give them a drag-and-drop interface … after an analyst prepares a data set via a . The process works by allowing the analyst to set up mappings between business logic and the raw underlying data. By exposing the high level business concepts to end users via the software’s interface, it enables them to do data exploration on their own without learning query languages like SQL. data modeling process This process started a few decades ago with vendors such as Cognos, SAP BI, and so on. The modern versions of this include vendors like Sisense, Tableau and PowerBI. However, regardless of old or new, these tools suffer from the same few problems. Legacy tools weren’t designed to take advantage of modern data warehouses. First, data is loaded into each vendor’s proprietary data store before becoming useful. This means data is now duplicated and that modern powerful MPP data warehouses becomes nothing more than a naive data store. The reason for this is simply because these tools weren’t designed to take advantage of modern data warehouses. They were designed at a time where data warehouses were expensive, and when it was natural for each BI tool to build their own datastore engines. Second, the data modeling process is mostly GUI-based, meaning it is not easy to reuse and can’t leverage powerful version control systems like git to track changes. The SQL-based data model approach The advantages of both approaches naturally point us to a new direction: what if we can combine the power of SQL and the ease of use of data modeling? The answer is that we absolutely can! Looker, which was recently acquired by Google for $2.6 billion, was a pioneer in using this approach. It provided a data modeling language called LookML which serves as an abstraction layer on top of a customer’s databases. The software does not load the data into its own data store or use a proprietary query engine. Instead, when a non-technical user uses Looker to explore data, the software translates the modeling input into a SQL query, and then sends this query to the customer’s database. This means the user doesn’t need to know SQL at all but still can utilize the power of modern MPP data warehouses! Another advantage of using a text-based modeling language is that now the whole modeling layer can be stored in a powerful version control system, making it trivial to track changes and to revert when there are issues. The SQL-based end-to-end data pipeline The SQL-based data model approach is just the tip of the iceberg. Take ETL, for instance. In order for our business data to be present in a data warehouse and ready to be modeled, one needs to import the data from multiple sources, and transform that data into a format that’s suitable for analysis. Traditionally, these tasks belong to a category of tools called ETL tools. The process includes three steps: Extract — extract data from sources. Transform — transform data to its final, usable format. Load — load that transformed data into the data warehouse. With the advent of powerful MPP data warehouses, more and more data teams are switching to an ELT approach, as opposed to an ETL approach. This process extracts raw data from sources and loads them directly into data warehouses with no transformations involved. Once stored, analysts can then use their favourite tool — SQL — to transform the data to a final format suitable for consumption. Because data transformations are now written in SQL, it follows naturally that we can apply the same SQL-based data modeling approach to it! Tools like Holistics take Looker’s SQL-based data modeling approach one step further by extending the modeling further into the L (load) and T (transform) layers, covering the entire analytics pipeline from end to end. This approach of using a unified SQL-based modeling approach to import, transform and explore data uncover multiple advantages that weren’t available before: It enables data analysts to , significantly reducing the time-to-insights of each analysis while keeping the requirements the same: that is, to the current industry standard, SQL. own the entire data analysis pipeline from end-to-end Data context is preserved. This means that the end results of an analysis can be traced back to where they come from, making it trivial to debug data accuracy issues. It such as data engineers, data analysts, data scientists and business users, thus improving the data culture of an organization. breaks the natural silos between data stakeholders The DataOps-driven movement When you start to think about data modeling as just another type of logical coding using SQL, the similarities between data analytics and software engineering became more and more apparent. The goal of software engineering is to deliver value by shipping software, while the goal of analytics is to deliver value by shipping explorable datasets and visualizations. Both of these processes include the need to write logical code (programming languages for the former and SQL modeling languages for the latter) then deploy the code to “production”. The software development world has seen multiple decades of process experimentation. One relatively modern movement that has been instrumental in improving software development is the DevOps movement. This movement incorporates automation and continuous delivery in order to help countless organizations speed up software delivery, while increasing reliability and quality of their software. My thesis is that we can significantly improve the data analytics process if we copy the ideas expressed in DevOps. To name a few examples: — If we treat analytics logic as code, we can now put them into our source code control systems, enabling powerful tracking, debugging, and (most importantly!) automation. Analytics as code — Data accuracy and quality can be tremendously improved by incorporating automated testing at every transformation step. Unit tests for data — Building a centralized data wiki allows everyone in an organization to discover and explore data in a single place. Centralized data wiki A new analytics paradigm SQL, data modeling and DevOps are three age-old concepts with little to do with one another. Yet when combined, we have an entirely new paradigm for thinking about analytics, one that’s never been seen before in our 60-year old industry. We call this the DataOps paradigm. It is the guiding paradigm behind Holistics, a unified analytics platform that we’ve been working on for the past two years. And we’re incredibly excited about its future. Originally published by Thanh, our Chief Engineer at https://www.holistics.io on November 15, 2019.

The Rise of Reusable SQL-based Data Modeling Tools and DataOps services

Una colección de los blogs más populares traducidos Traductores y editores: Carlos Secada , Felipe Chiriboga , Daniel Entin , Juan A. Navarro El mundo está recolectando más datos que nunca. Con ellos, puedes lograr cosas que habrían parecido imposibles hace sólo unos años atrás. La ciencia de datos es la disciplina de hacer que los datos sean útiles. Los tres sub campos de la Ciencia de Datos incluyen la extracción de grandes cantidades de información para la inspiración ( ), la toma de decisiones basada en información limitada ( ), y el uso de patrones en los datos para automatizar tareas ( ). analítica estadística Machine Learning / Inteligencia Artificial La educación en ciencia de datos contribuye en gran medida a garantizar que las personas sean los arquitectos de su propio brillante futuro. La Inteligencia Artificial (IA) representa un salto fundamental en el progreso humano y abre un futuro emocionante de oportunidades sin precedentes. Todos deben ser parte de ese futuro y participar en sus ideas, así que escribo para todos y no solo para los expertos. No quiero que las personas pierdan la oportunidad de hacer realidad sus sueños. ¡Con la ayuda y colaboración de traductores increíbles, también podemos derribar la barrera del idioma! Esta página te ofrece una colección de artículos en español para principiantes, ¡esperamos que los disfrutes! ******************************************************** La explicación más simple de Machine Learning que jamás habrás leído Seguramente has escuchado hablar de aprendizaje automático, o “machine learning” en inglés, y de inteligencia artificial. Pero, ¿en realidad sabes lo que son? Quizá no sabes bien de qué se tratan pero, te aseguro, no eres la única persona … La explicación más simple de Machine Learning que jamás habrás leido ¿Por qué las empresas fracasan haciendo Machine Learning? Imagina contratar a un chef para que te fabrique un horno, o un ingeniero eléctrico para hornear pan. Éste es el tipo de error que veo cometer a las empresas una y otra vez, cuando se trata de machine learning … ¿Por qué las empresas fracasan haciendo Machine Learning? ¿Qué diablos es Ciencia de Datos? Si exploramos la del nacimiento del término ciencia de datos, veremos dos temas que se juntan: (1) Big Data (Grandes Datos), que significa más trabajo para las computadoras. que pueden meter los pies en el hielo y la cabeza en el horno, y decir que en promedio están bien. historia Estadísticos Y así nace la ciencia de datos … ¿Qué diablos es Ciencia de Datos? Consejos para descubrir cuándo aplicar Inteligencia Artificial ¡Emocionate! El futuro está aquí: el aprendizaje automático, también conocido por sus siglas en inglés (ML — machine learning) y la inteligencia artificial (IA) te permiten automatizar lo inefable. Suena genial… si sabes cómo detectar un buen caso de uso … Imagina una isla borracha… Consejos para descubrir cuándo aplicar Inteligencia Artificial La verdadera razón por qué la Inteligencia Artificial es difícil No quiero que la AI sea sólo materia para académicos, expertos, grandes empresas y aquellos que hablan el mismo idioma que los bloggers de moda. Es mentira que la sea algo de genios y algo medio místico. Permíteme contarte las razones reales por las cuales es difícil utilizar AI hoy inteligencia artificial aplicada … La verdadera razón por qué la Inteligencia Artificial es difícil

Aprende sobre ciencia de datos e inteligencia artificial en español

This article is part 1 of a 5-part mini-series on DataOps at Alibaba . This installment looks at how Alibaba uses a project migration method to remove bottlenecks in its operations with big data platform MaxCompute, among other optimizations. In the world of big data, elaborate computing platforms are a necessary tool for dealing with the sheer scale of processing needs that users’ business goals present. More recently, however, they have also become the focus of efforts to evolve from human decision making toward data-driven decision making in operations and maintenance, creating a series of development challenges for major tech firms like e-commerce giant Alibaba. MaxCompute, formerly known as ODPS, stands alone in Alibaba’s technical ecosystem as an internally-developed big data platform carrying vast amounts of transaction data vital to the group’s operations. While this platform greatly facilitates the work of operations and maintenance professionals, it is nevertheless prone to performance-limiting congestion, requiring the group to devote considerable efforts toward optimization. Logically, MaxCompute is grouped as a unified big data pool, with transparent, interdependent EB-scale data. Physically, however, it is a cluster assembly spanning multiple regions, which presents a huge challenge in terms of capacity bottlenecks that can lead to the hinderances mentioned above. In MaxCompute, clusters contain multi-dimension resources such as computing, storage, and file count. They also have bandwidth resources for communications between clusters, with bandwidth costs varying according to the distance between clusters. Projects, meanwhile, differ in their consumption of resources in different dimensions, as well as in the time periods required for computing resource consumption. In addition, there are data access dependencies between projects that consume bandwidth resources between clusters. As business demands grow, the number and size of clusters likewise continue to expand, making the dependencies between projects increasingly complex. In light of these conditions, optimization is a process both of understanding and subsequently addressing a series of interdependent computing problems. In today’s article, we look more closely at the basis for these issues and the migration methods Alibaba employs specifically for its operations on MaxCompute, with insights aspiring data professionals can apply to their own work. Optimization Basics: Two Approaches When any of a cluster’s resources (computing, storage, or file count) reaches a bottleneck in MaxCompute, it affects the corresponding services and reduces the utility efficiency of other resources, causing capacity management problems. There are currently two main solutions to this issue. The first approach is capacity extension through machine procurement, which requires accurate forecasting of clusters’ future resource consumption and a reasonable procurement budget. The second is optimization of MaxCompute’s project layout across different clusters, including global layout optimization and local migration strategy optimization. Global layout optimization is equivalent to disorganizing all online projects and re-allocating them among clusters. Local migration strategy optimization involves project dispatching in advance for clusters prone to resource bottlenecks and stability issues. Due to restrictions from the degree of automation, migration efficiency, and actual business conditions surrounding project migration, implementing global layout optimization presents certain difficulties. To date, Alibaba has made significant progress in both procurement budgeting and local project migration. The following sections will introduce the limitations of previous approaches before explaining in detail how the data- and algorithm-based solution can be used to achieve local project migration strategies. Experience and Its Limits: Human-driven Solutions In the past, project migration work in MaxCompute relied heavily on the experience of Alibaba’s operation and maintenance personnel and their sensitivity to the “high water mark” for resources in each dimension. However, relying on human decision-making presents several limitations. First, since cluster resources span many dimensions, operation and maintenance personnel are liable to overlook migration needs in time to handle them, despite spending much of their time monitoring indicators. Another restriction is the high number of projects online, each of which has multi-dimensional attributes. Due to the differing intensity of data dependencies in different projects, it is difficult to consider all factors and make the optimal migration decisions. Operation and maintenance personnel usually develop a migration solution at a coarse granularity according to the category of the service. Finally, migration of a project will affect the entirety of MaxCompute at all levels, and it is difficult to accurately estimate and quantify this impact before the migration strategy has been fully implemented. In order to prevent a shortage of resources or a contraction in inter-cluster bandwidth following migration, migration plans generally err on the side of caution. Toward a data-driven solution: DataOps The resource dispatching experience that operation and maintenance personnel contribute is highly valuable. Still, big data platform construction and service providers need a weapon-like command of data and the ability to use it in decision-making to break through bottlenecks human insight alone cannot resolve. In this sense, data can help with rapid and even early detection of migration needs, refine solutions in project migration optimization strategy, and provide accurate quantitative assessment of migration results. Optimized Algorithm Design: A Blueprint for Migration Overall, data-based solutions to bottlenecking consists of three key steps. First, migration needs need to be determined and quantified at the cluster level. This decides which clusters and how many resources need to be migrated, which clusters will be able to receive the migrated project, and how much capacity each resource has. Second, linear programming is needed to obtain the optimal migration strategy. The objective of this step is to pick the most appropriate project to migrate to the most appropriate cluster. Lastly, quantitative estimation of the migration’s impact enables analysis of its consequences at the resource level and the bandwidth consumption of the cluster. Because real-life scenarios are complex and introduce problems such as cross-cluster replication, some impact factors can only be quantified with simulation computing according to the layout of the new project once the migration solution has been obtained. The purpose of quantifying the migration’s impact is to give operation and maintenance personnel a graphic, visual display which can help them judge the feasibility of the algorithm, which is especially important in its initial stage. You can also type a keyword to search online for the video that best fits your document. To make your document look professionally produced, Word provides header, footer, cover page, and text box designs that complement each other. Quantifying migration needs at the cluster level Each online cluster on MaxCompute has a different server size, computing and storage capacity, and scalability. Therefore, based on the expertise of operation and maintenance personnel and historical operation records of the clusters, Alibaba sets a custom “migration threshold” and “receiving threshold” for each resource of each cluster. Next, MaxCompute’s resources show obvious peak usage periods which are considered to help quantify migration needs, mainly in terms of the thresholds found during peak hours. Combined with the overall capacity of cluster resources, this yields an accurate sense of the specific migrating and receiving quantities. In addition, sometimes the demand presented by the clusters which need to be migrated exceeds the capacity of receiving clusters, indicating that the overall platform needs further capacity extension. However, since capacity procurement takes time, the algorithm adjusts migration needs according to the receiving capacity and generates a temporary migration solution to solve immediate online problems. Deploying linear programming After determining migration needs and receiving capabilities at the cluster level, the next step is to choose the best migration solution among the many projects in the clusters to be migrated. For this, there are several key points to consider. First, the solution must meet the migration needs presented by the computing, storage, and file count in the clusters to be migrated, and must not exceed the capacity of each corresponding resource in the receiving clusters. Secondly, unlike storage and file count, the consumption of computing resources of different projects depends highly on their time frames. Thus, computing resources need to be refined to hour-level granularity. Finally, as previously mentioned, there are data access dependencies between projects which consume bandwidth between clusters, with the bandwidth costs between clusters differing by regional location. In migrating projects, it is important to control bandwidth consumption between clusters. The process described above can be expressed as a constrained optimization problem, and both constraints and objective functions are linear. Linear programming is one of the best-known methods for solving optimization problems. The standard form of a typical linear programming is as follows: In the migration problem, a decision variable x can be used to determine whether a project should be migrated from cluster A to cluster B, with x=0 meaning that it should not be and x=1 meaning that it should. That is to say, the decision variable is a “0 or 1” variable, and therefore the linear programming problem here actually corresponds to 0/1 integer programming. In the objective function, the distance between data centers (the bandwidth cost) and the data dependencies between projects should be considered. Under the constraints, the migration needs for the resources of the clusters to be migrated and the capabilities of the receiving clusters are crucial information. There are many ways to solve linear programming problems, such as the simplex algorithm, interior point method, and many scientific calculation tools such as matlab and scipycan. There are also many optimization solvers that can efficiently solve large-scale linear programming, mixed integer programming, and other related problems, such as Cplex, Gurobi, and others. Migration’s Impact: Quantitative Estimation The impact of project migration on the cluster can help the operators to intuit the effect and impact of the migration more broadly, and can be divided into three parts for analysis. First, migration’s impact on the storage capacity level and file count should be considered. Since the data volume and number of files in a project are relatively stable, it is easy to determine the impact of the migration solution on the capacity level of the cluster’s storage and file count. Second, its impact on computing capacity levels must be examined. A project’s computing resource consumption has a strong correlation with time, and different projects vary greatly in their consumption of computing resources. The CPU consumption of each project in the migration list is amortized to every hour in the run-time period, and then the 24-hour CPU level of the cluster after migration is estimated. Lastly, the impact on traffic size between clusters needs to be analyzed. In the preceding linear programming, the data dependencies between projects are placed into the objective function for optimization, rather than into constraints. This is because according to the data dependencies between projects and the cluster layout of projects at the time, a cross-cluster replication list based on some complex rules is generated online each day, which converts the data that must be directly read across clusters into internally readable data within a cluster. Thus, there is no way to estimate the traffic between online clusters until the migration strategy has actually been created. After solving the migration strategy with linear programming, a new cross-cluster replication list needs to be simulated based on the rules and the new project layout, and then estimate the change in traffic between data centers brought by migration based on historical online traffic. Key Takeaways Smart migration systems allow operation and maintenance specialists to form an accurate quantitative assessment of each migration task, as well as a way to visualize the possible approaches they are aware of based on experience. With such platforms, they can not only see migration solutions dynamically computed according to the current capacity levels of clusters, but also input tailored migration requirements and constraints to customize an optimal migration strategy. Through parallel dispatch, they provide support for simultaneous operation and computation by multiple operation and maintenance engineers. For each migration requirement, the platform solves and displays the results in detail, including the specific optimal migration strategy obtained by the algorithm, the estimated changes in capacity level of the computing and storage cluster resources before and after migration, the estimated changes in inter-cluster traffic before and after migration, and the estimated changes in cross-region bandwidth usage before and after migration. Project migration optimization is a technology for offline big data services that uses DataOps to drive capacity management and resource allocation. In its algorithm model, multi-dimensional resource factors are integrated to find the optimal migration strategy under given constraints and to quantify the impact of migration. While already powerful, these tools still only reflect an early stage of development, and Alibaba continues to invest in optimization strategies that use machine learning and time series analysis methods to predict cluster resource consumption, guide machine procurement, and assist advanced detection of migration needs for early action on solutions. In our upcoming article, “The DataOps Files II: Resource Consumption”, we will look at how Alibaba is researching advanced forecasting models for future resource consumption. Alibaba Tech First hand and in-depth information about Alibaba’s latest technology → Facebook: . Twitter: . “Alibaba Tech” “AlibabaTech”

The DataOps Files I: Project Migration

Using Arrow Flight SQL Protocol in Apache Doris 2.1 For Super Fast Data Transfer

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk;
(2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk;
(3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction
Related work
Background
Methods
Experiments
"Conclusion, Limitations, and References"
Derivations
Algorithms
Experimental Datasets And Model Conclusion and Limitations We gave a new and far simpler method for computing posterior moments, marginals and samples in massively parallel importance sampling based on differentiating a slightly modified marginal likelihood estimator. The method has limitations, in that while it is considerably more effective than e.g. VI, HMC and global importance sampling, it is more complex. Additionally, at least a naive implementation may be quite costly in terms of memory consumption, limiting how the number of importance samples we can draw for each variable. That said, it should be possible to eliminate almost all of this overhead by careful optimizations to avoid allocating large intermediate tensors, following the strategy in KeOps (Charlier et al. 2021). References Aitchison, L. 2019. Tensor Monte Carlo: particle methods for the GPU era. Advances in Neural Information Processing Systems, 32. Albarghouthi, A.; D’Antoni, L.; Drews, S.; and Nori, A. V. 2017. Fairsquare: probabilistic verification of program fairness. Proceedings of the ACM on Programming Languages, 1(OOPSLA): 1–30. Andrieu, C.; Doucet, A.; and Holenstein, R. 2010. Particle markov chain monte carlo methods. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 72(3): 269–342. Bidyuk, B.; and Dechter, R. 2007. Cutset sampling for Bayesian networks. Journal of Artificial Intelligence Research, 28: 1–48. Bornschein, J.; and Bengio, Y. 2014. Reweighted wakesleep. arXiv preprint arXiv:1406.2751. Bradbury, J.; Frostig, R.; Hawkins, P.; Johnson, M. J.; Leary, C.; Maclaurin, D.; Necula, G.; Paszke, A.; VanderPlas, J.; Wanderman-Milne, S.; and Zhang, Q. 2018. JAX: composable transformations of Python+NumPy programs. Burda, Y.; Grosse, R.; and Salakhutdinov, R. 2015. Importance weighted autoencoders. arXiv preprint arXiv:1509.00519. Charlier, B.; Feydy, J.; Glaunes, J. A.; Collin, F.-D.; and Du- ` rif, G. 2021. Kernel Operations on the GPU, with Autodiff,  without Memory Overflows. Journal of Machine Learning Research. Chatterjee, S.; and Diaconis, P. 2018. The sample size required in importance sampling. The Annals of Applied Probability, 28(2): 1099–1135. Claret, G.; Rajamani, S. K.; Nori, A. V.; Gordon, A. D.; and Borgstrom, J. 2013. Bayesian inference using data flow anal- ¨ ysis. In Proceedings of the 2013 9th joint meeting on foundations of software engineering, 92–102. Corenflos, A.; Chopin, N.; and Sarkk ¨ a, S. 2022. De- ¨ Sequentialized Monte Carlo: a parallel-in time particle smoother. arXiv preprint arXiv:2202.02264. Crucinio, F. R.; and Johansen, A. M. 2023. Properties of Marginal Sequential Monte Carlo Methods. ArXiv:2303.03498 [math, stat]. Dawid, A. P. 1992. Applications of a general propagation algorithm for probabilistic expert systems. Statistics and computing, 2(1): 25–36. Dechter, R.; Broka, F.; Kask, K.; and Ihler, A. 2018. Abstraction Sampling in Graphical Models. AAAI. Del Moral, P.; Doucet, A.; and Jasra, A. 2006. Sequential Monte Carlo Samplers. Journal of the Royal Statistical Society Series B: Statistical Methodology, 68(3): 411–436. DOE. 2023. Bus Breakdown and Delays. url https://data.cityofnewyork.us/Transportation/BusBreakdown-and-Delays/ez4e-fazm Accessed on: 05.05.2023. Doucet, A.; Johansen, A. M.; et al. 2009. A tutorial on particle filtering and smoothing: Fifteen years later. Handbook of nonlinear filtering, 12(656-704): 3. Geffner, T.; and Domke, J. 2022. Variational Inference with Locally Enhanced Bounds for Hierarchical Models. arXiv preprint arXiv:2203.04432. Gehr, T.; Misailovic, S.; and Vechev, M. 2016. PSI: Exact symbolic inference for probabilistic programs. In International Conference on Computer Aided Verification, 62–83. Springer. Geldenhuys, J.; Dwyer, M. B.; and Visser, W. 2012. Probabilistic symbolic execution. In Proceedings of the 2012 International Symposium on Software Testing and Analysis, 166–176. Gogate, V.; and Dechter, R. 2012. Importance samplingbased estimation over and/or search spaces for graphical models. Artificial Intelligence, 184: 38–77. Goodman, N. D.; and Stuhlmuller, A. 2014. The design and implementation of probabilistic programming languages. Gordon, N. J.; Salmond, D. J.; and Smith, A. F. 1993. Novel approach to nonlinear/non-Gaussian Bayesian state estimation. In IEE proceedings F (radar and signal processing), volume 140, 107–113. IET. Harper, F. M.; and Konstan, J. A. 2015. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis), 5(4): 1–19. Heap, T.; and Laurence, A. 2023. Massively parallel reweighted wake-sleep. UAI. Hoffman, M. D.; Gelman, A.; et al. 2014. The No-UTurn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo. J. Mach. Learn. Res., 15(1): 1593–1623. Holtzen, S.; Van den Broeck, G.; and Millstein, T. 2020. Scaling exact inference for discrete probabilistic programs. Proceedings of the ACM on Programming Languages, 4(OOPSLA): 1–31. Jordan, M. I.; Ghahramani, Z.; Jaakkola, T. S.; and Saul, L. K. 1999. An introduction to variational methods for graphical models. Machine learning, 37(2): 183–233. Kingma, D. P.; Welling, M.; et al. 2019. An introduction to variational autoencoders. Foundations and Trends® in Machine Learning, 12(4): 307–392. Kuntz, J.; Crucinio, F. R.; and Johansen, A. M. 2023. The divide-and-conquer sequential Monte Carlo algorithm: theoretical properties and limit theorems. ArXiv:2110.15782 [math, stat]. Lai, J.; Domke, J.; and Sheldon, D. 2022. Variational Marginal Particle Filters. In International Conference on Artificial Intelligence and Statistics, 875–895. PMLR. Le, T. A.; Igl, M.; Rainforth, T.; Jin, T.; and Wood, F. 2017. Auto-encoding sequential monte carlo. arXiv preprint arXiv:1705.10306. Le, T. A.; Kosiorek, A. R.; Siddharth, N.; Teh, Y. W.; and Wood, F. 2020. Revisiting reweighted wake sleep for models with stochastic control flow. In Uncertainty in Artificial Intelligence, 1039–1049. PMLR. Lindsten, F.; Johansen, A. M.; Naesseth, C. A.; Kirkpatrick, B.; Schon, T. B.; Aston, J.; and Bouchard-Cotˆ e, A. 2017. ´ Divide-and-conquer with sequential Monte Carlo. Journal of Computational and Graphical Statistics, 26(2): 445–458. Maddison, C. J.; Lawson, J.; Tucker, G.; Heess, N.; Norouzi, M.; Mnih, A.; Doucet, A.; and Teh, Y. 2017. Filtering variational objectives. Advances in Neural Information Processing Systems, 30. Naesseth, C.; Linderman, S.; Ranganath, R.; and Blei, D. 2018. Variational sequential monte carlo. In International conference on artificial intelligence and statistics, 968–977. PMLR. Narayanan, P.; Carette, J.; Romano, W.; Shan, C.-c.; and Zinkov, R. 2016. Probabilistic inference by program transformation in Hakaru (system description). In International Symposium on Functional and Logic Programming, 62–79. Springer Obermeyer, F.; Bingham, E.; Jankowiak, M.; Pradhan, N.; Chiu, J.; Rush, A.; and Goodman, N. 2019. Tensor variable elimination for plated factor graphs. In International Conference on Machine Learning, 4871–4880. PMLR. Pfeffer, A. 2005. The design and implementation of IBAL: A generalpurpose probabilistic programming language. In Harvard Univesity. Citeseer. Salvatier, J.; Wiecki, T. V.; and Fonnesbeck, C. 2016. Probabilistic programming in Python using PyMC3. PeerJ Computer Science, 2: e55. Publisher: PeerJ Inc. Sankaranarayanan, S.; Chakarov, A.; and Gulwani, S. 2013. Static analysis for probabilistic programs: inferring whole program properties from finitely many paths. In Proceedings of the 34th ACM SIGPLAN conference on Programming language design and implementation, 447–458. Wang, D.; Hoffmann, J.; and Reps, T. 2018. PMAF: an algebraic framework for static analysis of probabilistic programs. ACM SIGPLAN Notices, 53(4): 513–528. Weinberg, S. 1995. The quantum theory of fields, volume 2. Cambridge university press. Zavatone-Veth, J.; Canatar, A.; Ruben, B.; and Pehlevan, C. 2021. Asymptotics of representation learning in finite Bayesian neural networks. Advances in neural information processing systems, 34: 24765–24777. This paper is available on arxiv under CC 4.0 license. Authors: (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. This paper is available on arxiv under CC 4.0 license. Authors: Authors: (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction Related work Background Methods Experiments "Conclusion, Limitations, and References" Derivations Algorithms Experimental Datasets And Model Abstract & Introduction Abstract & Introduction Related work Related work Background Background Methods Methods Experiments Experiments "Conclusion, Limitations, and References" "Conclusion, Limitations, and References" Derivations Derivations Algorithms Algorithms Experimental Datasets And Model Experimental Datasets And Model Conclusion and Limitations We gave a new and far simpler method for computing posterior moments, marginals and samples in massively parallel importance sampling based on differentiating a slightly modified marginal likelihood estimator. The method has limitations, in that while it is considerably more effective than e.g. VI, HMC and global importance sampling, it is more complex. Additionally, at least a naive implementation may be quite costly in terms of memory consumption, limiting how the number of importance samples we can draw for each variable. That said, it should be possible to eliminate almost all of this overhead by careful optimizations to avoid allocating large intermediate tensors, following the strategy in KeOps (Charlier et al. 2021). References Aitchison, L. 2019. Tensor Monte Carlo: particle methods for the GPU era. Advances in Neural Information Processing Systems, 32. Albarghouthi, A.; D’Antoni, L.; Drews, S.; and Nori, A. V. 2017. Fairsquare: probabilistic verification of program fairness. Proceedings of the ACM on Programming Languages, 1(OOPSLA): 1–30. Andrieu, C.; Doucet, A.; and Holenstein, R. 2010. Particle markov chain monte carlo methods. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 72(3): 269–342. Bidyuk, B.; and Dechter, R. 2007. Cutset sampling for Bayesian networks. Journal of Artificial Intelligence Research, 28: 1–48. Bornschein, J.; and Bengio, Y. 2014. Reweighted wakesleep. arXiv preprint arXiv:1406.2751. Bradbury, J.; Frostig, R.; Hawkins, P.; Johnson, M. J.; Leary, C.; Maclaurin, D.; Necula, G.; Paszke, A.; VanderPlas, J.; Wanderman-Milne, S.; and Zhang, Q. 2018. JAX: composable transformations of Python+NumPy programs. Burda, Y.; Grosse, R.; and Salakhutdinov, R. 2015. Importance weighted autoencoders. arXiv preprint arXiv:1509.00519. Charlier, B.; Feydy, J.; Glaunes, J. A.; Collin, F.-D.; and Du- ` rif, G. 2021. Kernel Operations on the GPU, with Autodiff,  without Memory Overflows. Journal of Machine Learning Research. Chatterjee, S.; and Diaconis, P. 2018. The sample size required in importance sampling. The Annals of Applied Probability, 28(2): 1099–1135. Claret, G.; Rajamani, S. K.; Nori, A. V.; Gordon, A. D.; and Borgstrom, J. 2013. Bayesian inference using data flow anal- ¨ ysis. In Proceedings of the 2013 9th joint meeting on foundations of software engineering, 92–102. Corenflos, A.; Chopin, N.; and Sarkk ¨ a, S. 2022. De- ¨ Sequentialized Monte Carlo: a parallel-in time particle smoother. arXiv preprint arXiv:2202.02264. Crucinio, F. R.; and Johansen, A. M. 2023. Properties of Marginal Sequential Monte Carlo Methods. ArXiv:2303.03498 [math, stat]. Dawid, A. P. 1992. Applications of a general propagation algorithm for probabilistic expert systems. Statistics and computing, 2(1): 25–36. Dechter, R.; Broka, F.; Kask, K.; and Ihler, A. 2018. Abstraction Sampling in Graphical Models. AAAI. Del Moral, P.; Doucet, A.; and Jasra, A. 2006. Sequential Monte Carlo Samplers. Journal of the Royal Statistical Society Series B: Statistical Methodology, 68(3): 411–436. DOE. 2023. Bus Breakdown and Delays. url https://data.cityofnewyork.us/Transportation/BusBreakdown-and-Delays/ez4e-fazm Accessed on: 05.05.2023. Doucet, A.; Johansen, A. M.; et al. 2009. A tutorial on particle filtering and smoothing: Fifteen years later. Handbook of nonlinear filtering, 12(656-704): 3. Geffner, T.; and Domke, J. 2022. Variational Inference with Locally Enhanced Bounds for Hierarchical Models. arXiv preprint arXiv:2203.04432. Gehr, T.; Misailovic, S.; and Vechev, M. 2016. PSI: Exact symbolic inference for probabilistic programs. In International Conference on Computer Aided Verification, 62–83. Springer. Geldenhuys, J.; Dwyer, M. B.; and Visser, W. 2012. Probabilistic symbolic execution. In Proceedings of the 2012 International Symposium on Software Testing and Analysis, 166–176. Gogate, V.; and Dechter, R. 2012. Importance samplingbased estimation over and/or search spaces for graphical models. Artificial Intelligence, 184: 38–77. Goodman, N. D.; and Stuhlmuller, A. 2014. The design and implementation of probabilistic programming languages. Gordon, N. J.; Salmond, D. J.; and Smith, A. F. 1993. Novel approach to nonlinear/non-Gaussian Bayesian state estimation. In IEE proceedings F (radar and signal processing), volume 140, 107–113. IET. Harper, F. M.; and Konstan, J. A. 2015. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis), 5(4): 1–19. Heap, T.; and Laurence, A. 2023. Massively parallel reweighted wake-sleep. UAI. Hoffman, M. D.; Gelman, A.; et al. 2014. The No-UTurn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo. J. Mach. Learn. Res., 15(1): 1593–1623. Holtzen, S.; Van den Broeck, G.; and Millstein, T. 2020. Scaling exact inference for discrete probabilistic programs. Proceedings of the ACM on Programming Languages, 4(OOPSLA): 1–31. Jordan, M. I.; Ghahramani, Z.; Jaakkola, T. S.; and Saul, L. K. 1999. An introduction to variational methods for graphical models. Machine learning, 37(2): 183–233. Kingma, D. P.; Welling, M.; et al. 2019. An introduction to variational autoencoders. Foundations and Trends® in Machine Learning, 12(4): 307–392. Kuntz, J.; Crucinio, F. R.; and Johansen, A. M. 2023. The divide-and-conquer sequential Monte Carlo algorithm: theoretical properties and limit theorems. ArXiv:2110.15782 [math, stat]. Lai, J.; Domke, J.; and Sheldon, D. 2022. Variational Marginal Particle Filters. In International Conference on Artificial Intelligence and Statistics, 875–895. PMLR. Le, T. A.; Igl, M.; Rainforth, T.; Jin, T.; and Wood, F. 2017. Auto-encoding sequential monte carlo. arXiv preprint arXiv:1705.10306. Le, T. A.; Kosiorek, A. R.; Siddharth, N.; Teh, Y. W.; and Wood, F. 2020. Revisiting reweighted wake sleep for models with stochastic control flow. In Uncertainty in Artificial Intelligence, 1039–1049. PMLR. Lindsten, F.; Johansen, A. M.; Naesseth, C. A.; Kirkpatrick, B.; Schon, T. B.; Aston, J.; and Bouchard-Cotˆ e, A. 2017. ´ Divide-and-conquer with sequential Monte Carlo. Journal of Computational and Graphical Statistics, 26(2): 445–458. Maddison, C. J.; Lawson, J.; Tucker, G.; Heess, N.; Norouzi, M.; Mnih, A.; Doucet, A.; and Teh, Y. 2017. Filtering variational objectives. Advances in Neural Information Processing Systems, 30. Naesseth, C.; Linderman, S.; Ranganath, R.; and Blei, D. 2018. Variational sequential monte carlo. In International conference on artificial intelligence and statistics, 968–977. PMLR. Narayanan, P.; Carette, J.; Romano, W.; Shan, C.-c.; and Zinkov, R. 2016. Probabilistic inference by program transformation in Hakaru (system description). In International Symposium on Functional and Logic Programming, 62–79. Springer Obermeyer, F.; Bingham, E.; Jankowiak, M.; Pradhan, N.; Chiu, J.; Rush, A.; and Goodman, N. 2019. Tensor variable elimination for plated factor graphs. In International Conference on Machine Learning, 4871–4880. PMLR. Pfeffer, A. 2005. The design and implementation of IBAL: A generalpurpose probabilistic programming language. In Harvard Univesity. Citeseer. Salvatier, J.; Wiecki, T. V.; and Fonnesbeck, C. 2016. Probabilistic programming in Python using PyMC3. PeerJ Computer Science, 2: e55. Publisher: PeerJ Inc. Sankaranarayanan, S.; Chakarov, A.; and Gulwani, S. 2013. Static analysis for probabilistic programs: inferring whole program properties from finitely many paths. In Proceedings of the 34th ACM SIGPLAN conference on Programming language design and implementation, 447–458. Wang, D.; Hoffmann, J.; and Reps, T. 2018. PMAF: an algebraic framework for static analysis of probabilistic programs. ACM SIGPLAN Notices, 53(4): 513–528. Weinberg, S. 1995. The quantum theory of fields, volume 2. Cambridge university press. Zavatone-Veth, J.; Canatar, A.; Ruben, B.; and Pehlevan, C. 2021. Asymptotics of representation learning in finite Bayesian neural networks. Advances in neural information processing systems, 34: 24765–24777.

Using Autodiff to Estimate Posterior Moments: Conclusion, Limitations, and References

B2B Tools You Need To Have in Your Arsenal in 2024

Hello everyone and thanks for dropping by. Today I would like to explore how one might go about creating their own sitemap generator script for a static Next.js website. Of course there are packages out there that do just this, but there is great value in writing your own and understanding the solution fully. I'll assume that readers have some familiarity with Next.js, JavaScript, Node.js and SEO to set the tone for what follows. In addition, I assume the reader has a static Next.js website or can create one to follow along with the post. Ok, now let's get started. Background The motivation for this post really came from some work I was doing on my personal website: . The goal of the site is to be an engaging and educational blog that is communicated in a digestible way for any reader. I wanted to customize the blog as much as I'd like with little to no cost and reacquaint myself with React.js. I initially thought a vanilla React.js application would be great, but SPAs have their limitations when it comes to SEO if you aren't server side rendering your pages. So I chose to host a static Next.js site on Google Cloud Storage to make my content indexable by search engines. afro-cloud.com Let's get refocused on the concept of sitemaps. You can think of sitemaps as a list of pages for your website that help search engines understand the contents of your website. In short, they tell search engine crawlers what contents on your site are important to you. But sitemaps aren't just limited to pages, they can be used to indicate other types of content on your site like video for example. So if you want your pages to be indexed by a search engine like Google, it's important to indicate to their crawlers what content matters to you. The sitemap.xml file conveys this information and we will be building a script to generate one for a static Next.js website. Understanding Next.js Routing Next.js has two different routing paradigms: "Using app router" and "Using page router". For the purposes of this post, we will be using the app router approach. The TLDR is that all pages for our website will be contained within a top-level folder called app/. But within app/, we can have nested directories to represent the hierarchy of pages for our site. The way we indicate to Next.js that we have pages for our website is by having a page.jsx or page.tsx file within those directories. Take this directory structure for our app represented in a nested list for example: /app page.tsx /about page.tsx /blog page.tsx /post1 page.tsx /post2 page.tsx For this example we have the following page paths: /app/page.tsx /app/about/page.tsx /app/blog/page.tsx /app/blog/post1/page.tsx /app/blog/post2/page.tsx For sake of example, let's say we're building a website with the domain: wethebest.com. If we translate the above list to client routes we get: wethebest.com wethebest.com/about wethebest.com/blog wethebest.com/blog/post1 wethebest.com/blog/post2 Since we are using a static website approach, we need to map our client side routes to the fully qualified path of the HTML files corresponding to each page to ensure refreshes navigate to the same page. Doing so yields the following: wethebest.com wethebest.com/about.html wethebest.com/blog.html wethebest.com/blog/post1.html wethebest.com/blog/post2.html Ok, so now we have some pages, but what does Next.js offer to create a sitemap.xml for our application? Next.js provides , but you have to define a configuration object that corresponds to what you want in your sitemap.xml. But we can do better. I for one am lazy and don't want to add a new entry to a configuration every time I add a page to my site. Let's create our own script to generate a sitemap for us. some support Writing a sitemap.xml Generator Script So how might you start to tackle this problem of generating our own sitemap.xml generator script? We know how the client side routing works in Next.js for our use case. We know that our sitemap.xml URL entries need to have the static HTML file included in the URL for each page. We know that whenever we add a new page to our site, we want the URL of that page to be reflected in our sitemap.xml. Hopefully you're thinking about reading the content of our app directory and using that information to build our sitemap. If not, no worries! We'll go through it together. Let's first start by creating a file at the root of our project alongside our . The contents of the file will be the following to start: sitemap.js package.json const fs = require("node:fs");
const { SitemapStream, streamToPromise } = require("sitemap");
const { Readable } = require("stream");

const SITEMAP_HOSTNAME = "https://www.wethebest.com";

(async () => {
// Create [SitemapStream] that will ultimately populate our sitemap.xml.
const stream = new SitemapStream({
hostname: SITEMAP_HOSTNAME,
});
})(); We'll be using the Node.js file system sync API to read the contents of our directories. We'll also be using an NPM package called sitemap that provides utilities to help make creating our sitemap easy. In order to take advantage of the sitemap package, we'll need to create a readable stream that Node.js provides. We have a constant variable declaration that we'll use to build our URLs for the sitemap.xml file. Finally, we have an immediately invoked function expression which will be executed whenever we execute this file. Now let's add the code to start traversing our app/ directory and find the pages for our website. The first thing we'll want to do is start exploring our top-level app/ folder and find all of our pages file paths. The reason we are doing this is that we can map the file path to the corresponding HTML files for our site (look back at the routing section if you need a refresher). We can use the following code to do that Update your sitemap.js file to look like this: const fs = require("node:fs");
const { SitemapStream, streamToPromise } = require("sitemap");
const { Readable } = require("stream");

const PAGE_FILENAME = "page.tsx";
const SITEMAP_HOSTNAME = "https://www.wethebest.com";
const SITEMAP_PATHS = [];

Populates [SITEMAP_PATHS] with the fully qualified URLs for our pages to be contained

within our sitemap.xml file.

@param {string} path The path we are currently searching.

@param {string} isRoot Flag that denotes whether or not we are processing our root path.

@return {void}
*/
const getAllUrlsForSitemap = (path, isRoot) => {
// Get all files within our current directory
const files = fs.readdirSync(path);
files.forEach((file) => {
// Not using template strings due to article format on website.
const dirPath = 'path' + '/' + 'file';
if (file.includes(PAGE_FILENAME) && !isRoot) {
SITEMAP_PATHS.push(dirPath);
} else {
try {
const stats = fs.statSync(dirPath);
if (stats.isDirectory()) {
// Continue searching until we exhaust all directories.
getAllUrlsForSitemap(dirPath);
}
} catch (err) {}
}
});
};

(async () => {
// Create [SitemapStream] that will ultimately populate our sitemap.xml.
const stream = new SitemapStream({
hostname: SITEMAP_HOSTNAME,
});

// Populate [SITEMAP_PATHS] based on our [PAGE_FILENAME].
getAllUrlsForSitemap(APP_ROOT, true);
})(); You'll notice that we have a new method declaration: getAllUrlsForSitemap. This method navigates our app/ directory contents and searches for page.tsx files as well as other directories. We have a special case for our root page (app/page.tsx) because we'll add that entry later. Since we have some nested paths, we will need to explore more than the top-level app folder. The output of the readdirSync gives us all the contents of the directory and we loop through the contents to check for our pages and nested directories. Finally, when we find a page, we add the corresponding path to a variable SITEMAP_PATHS that we will use for processing later on. Now let's tie it all together. We have all of the paths for the pages we need to generate our sitemap, now all we have to do is map those to the HTML files and create the sitemap. Update your sitemap.js file to look like this: const fs = require("node:fs");
const { SitemapStream, streamToPromise } = require("sitemap");
const { Readable } = require("stream");

const APP_ROOT = "/Users/claudioherrera/Documents/GitHub/wethebest.com/app";
const PAGE_FILENAME = "page.tsx";
const SITEMAP_HOSTNAME = "https://www.wethebest.com";
const SITEMAP_PATHS = [];
const TODAYS_DATE_AS_ARRAY = new Date().toLocaleDateString().split("/");
const WRITE_FILE_PATH = "./app/sitemap.xml";

/**

Populates [SITEMAP_PATHS] with the fully qualified URLs for our pages to be contained

within our sitemap.xml file.

@param {string} path The path we are currently searching.

@param {string} isRoot Flag that denotes whether or not we are processing our root path.

@return {void}
*/
const getAllUrlsForSitemap = (path, isRoot) => {
// Get all files within our current directory
const files = fs.readdirSync(path);
files.forEach((file) => {
// Not using template strings due to article format on website.
const dirPath = 'path' + '/' + 'file';
if (file.includes(PAGE_FILENAME) && !isRoot) {
SITEMAP_PATHS.push(dirPath);
} else {
try {
const stats = fs.statSync(dirPath);
if (stats.isDirectory()) {
// Continue searching until we exhaust all directories.
getAllUrlsForSitemap(dirPath);
}
} catch (err) {}
}
});
};

/**

Returns an object containing the url and lastmod sitemap properties.

@param {string} path The path we are currently searching.

@param {string} isRoot Flag that denotes whether or not we are processing our root path.

@return {Object} url: URL for page on website. lastmod: last modification date for our page.
*/
const mapToSitemapFormat = (path, lastmod) => ({
// Not using template strings due to article format on website.
url: path
.replace(APP_ROOT + '/', "")
.replace('/' + PAGE_FILENAME, "") + '.html',
lastmod,
});

/**

Returns a string representation of a date unit. e.g. dd, mm, yy

@param {string} unit Date unit as a number in string format. e.g. month, day, year

@param {string} isRoot Flag that denotes whether or not we are processing our root path.

@return {Object} url: URL for page on website. lastmod: last modification date for our page.
*/
const padDateUnit = (unit) => ('unit'.length < 2 ? '0' + 'unit' : unit);

(async () => {
console.info("Generating sitemap...");

// Create [SitemapStream] that will ultimately populate our sitemap.xml.
const stream = new SitemapStream({
hostname: SITEMAP_HOSTNAME,
});

// Populate [SITEMAP_PATHS] based on our [PAGE_FILENAME].
getAllUrlsForSitemap(APP_ROOT, true);

try {
// Create our sitemap.xml contents. Not using template strings due to article format on website.
const LAST_MODIFIED_DATE = TODAYS_DATE_AS_ARRAY[2] + '-' + padDateUnit(
TODAYS_DATE_AS_ARRAY[0]
) + '-' + padDateUnit(TODAYS_DATE_AS_ARRAY[1]);
const ROOT_SITEMAP_ENTRY = {
url: SITEMAP_HOSTNAME,
lastmod: LAST_MODIFIED_DATE,
};
const content = await streamToPromise(
Readable.from([
ROOT_SITEMAP_ENTRY,
...SITEMAP_PATHS.map((url) =>
mapToSitemapFormat(url, LAST_MODIFIED_DATE)
),
]).pipe(stream)
).then((data) => data.toString());

console.info("Successfully generated sitemap...");

// Write our sitemap contents to [WRITE_FILE_PATH].
fs.writeFileSync(WRITE_FILE_PATH, content);

} catch (err) {
console.error(err);
}
})(); The core of the difference between this snippet and the one above is that we have now tied together the last two pieces to generate our sitemap. We are mapping our SITEMAP_PATHS to the sitemap.xml format by creating objects that contain a url and lastmod key. These objects are then fed into the sitemap package to generate the sitemap in XML format. Finally, we write the sitemap.xml to the location of our choosing. Let's execute the sitemap.js script by running "node sitemap.js" (note you will need to have Node.js installed to be in the current working directory of the sitemap file). You should now see a sitemap.xml file in your app folder and it should look something like: <?xml version="1.0" encoding="UTF-8"?>
<urlset
xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:news="http://www.google.com/schemas/sitemap-news/0.9"
xmlns:xhtml="http://www.w3.org/1999/xhtml"
xmlns:image="http://www.google.com/schemas/sitemap-image/1.1"
xmlns:video="http://www.google.com/schemas/sitemap-video/1.1">
<url>
<loc>https://www.wethebest.com/</loc>
<lastmod>2024-03-10T00:00:00.000Z</lastmod>
</url>
<url>
<loc>https://www.wethebest.com/about.html</loc>
<lastmod>2024-03-10T00:00:00.000Z</lastmod>
</url>
<url>
<loc>https://www.wethebest.com/blog.html</loc>
<lastmod>2024-03-10T00:00:00.000Z</lastmod>
</url>
<url>
<loc>https://www.wethebest.com/blog/post1.html</loc>
<lastmod>2024-03-10T00:00:00.000Z</lastmod>
</url>
<url>
<loc>https://www.wethebest.com/blog/post2.html</loc>
<lastmod>2024-03-10T00:00:00.000Z</lastmod>
</url>
</urlset> Nice! Now, whenever you add a new page to your Next.js app, you can easily include it in your sitemap. I highly recommend adding a new production build rule to your application to incorporate this sitemap generation whenever you deploy a new version of your website. I hope you found this post useful. Cheers! Also appears . here

Stories, News & Mentions about DAATO

Daato’s Stories on HackerNoon