Castor

castor

"If I change this table, how can i know I won't break any tables downstream?" "If I replace this table with another better designed, what are all the dashboards I need to update with the new data table?" "This dashboard seems wrong. Where can I check what data powers this dashboard?" I can't say how many times I've asked myself these questions, or how many times I heard those when I talk with data engineers, analytics engineers or heads of data. The answer never was rally exciting. It was something like: "well, check out the source code and map dependencies in an excel" or "ask Brian, he knows". Obviously, that poor Brian was an incredible data engineer, but the reason he was so good was that he wasn't spending his time answering to my relentless pings on Slack. : that's why people invented data lineage ! Spoiler Alert 'Data lineage is like a family tree but for data' Data lineage is a technology that retraces the relationships between data assets. In the data world, you start by collecting raw data from various sources (logs from your website, payments, etc) and refine this data by applying successive transformations. In order to build one data table (we will call it "child table"), you have to use one or more other data tables (we call these "parent tables"). The data lineage helps you rebuild the family tree of your data. If you have bad data, you can look for the 'bad branch' in that data family tree and cure it to the roots. In the graph below, each card represents a data resources like a table, a dashboard or a report. Each link represents a transformation or a data job (orchestrated by Airflow for example). 'Data lineage is like a family tree but for data' What is the difference between Table and Column-level lineage? Data lineage flow can exist under different format. You can either trace parent-child relationships between data tables or columns in a specific table. Indeed, each column is created with other columns from other tables, sometimes even from the same table. Column lineage is a subset of table lineage. If there is a link between two columns from different tables, then there is a link between the two tables. The opposite isn't always true. Column lineage is providing the parent-child relationship at a column-level without taking into account the tables underneath. It is a really powerful kind of lineage but can quickly be extremely hard to visualize as the number of connections grows. Column-level lineage system is also particularly hard to compute. Processing column-lineage more expensive, takes longer and requires a finer version of the parsing algorithm to understand the changes. Example of what column-lineage flow looks like in Castor . Table-level approach In the SELECT SQL statement below, we are building a table with two other tables (FROM clause) and (JOIN clause). That means that if there is a change or a problem in one of the two "parent tables", the child table might be affected by the change or the bug. child_table_1 parent_table_1 parent_table_2 Example of SQL query that can be parsed to find parent and column-level lineage links. Column-level approach In the SELECT SQL statement above, we are building three columns , , and in the . The two first "child columns" are the same as the "parent columns" from and . There is a direct lineage dependence. child_1 child_2 child_3 child_table_1 parent_table_1 parent_table_2 The column is a sum of and . There's a lineage dependence here as well. child_3 parent_table_1.parent_column1 parent_table_1.parent_column1 "Data lineage gives a bird's eye view of a data flow." It helps trace back relationships across data systems A few years ago, lineage was mostly about tracing relationships in the data lake or the data warehouse. This means a system tracking relationships between data tables. Now data lineage is cross-system. Those tables are often powering BI tools such as Looker, Tableau, Mode, Metabase, PowerBI etc. Mapping this flow proved really powerful for business users as well as engineers. Indeed, business users can fact-check the provenance and flow of the data easily while engineers can check the impact of a change on the business. End-to-end data lineage system from the data sources to the BI tools. End-to-end lineage, from data sources (ETL) to business reports, is possible now that most BI tools have APIs, and that BI tools perform SQL queries on the data warehouse in a standardized way. When data catalog providers had to build as many versions of their lineage parser for each new tool, language and integration, they can now build a lineage engine and wait customize the last mile connector. Why do we need data lineage? Data lineage use-cases First of all, data lineage is a technology more than a product. It is one of the underlying technologies behind a lot of data products (data catalogs, data quality, data management, etc). The reason data lineage is so popular is that there are a lot of new use-cases, both for business, engineering, leadership, and legal department. Let me try to explain the most common use-cases. Overall, data lineage helps get a bird's eye view of the data systems. Data Troubleshooting End-to-end data lineage is extremely powerful to troubleshoot data problems. Data users sometimes spot weird behavior in dashboards or data tables. They want to investigate to know if this is due to a technical problem in the data pipelines or a business problem that needs attention right now. There's nothing more humiliating for a data person to sound the alarm about a business problem (for example, a drop in revenue for the "active users" segment) when it really is a data pipeline that broke. In the example above, the data in the dashboard drops drastically. The business user wants to understand why this happened. The underlying question is: "do I have a business problem or a data pipeline problem?". He can check the status of all upstream data. In this case, the problem comes from a data source (the app store) that changed its API. This change broke the ETL job feeding data sources. In a few minutes, the user can understand the data system as a whole. Impact analysis Data lineage helps you visualize the impact of a change Data users want to know the impact of a change they want to make on the data flow, and on data resources downstream. In most organizations, data reports and dashboards are powered by data tables in the data lake. If an engineer makes a change, this will likely break data pipelines and processes. As a result, overall data quality and trust within the organization will drastically decrease. Thanks to the lineage, engineers can identify all impacted resources and craft their code to minimize bad impact. If there is an impact, they can identify the data resources affected and send an warning to active users of those resources. Discovery and Trust Data Lineage helps you trust your data by knowing where it comes from Users want to identify, understand and trust data resources (tables, reports, and dashboards) that they want to consume. When a user looks at data resources, knowing where it comes from helps to trust it. A saying says "garbage in, garbage out". Knowing what is the data powering the dashboard on "active users" will give you information on how to read it and if you can trust it. Lineage will let you know that it is plugged to an "Approved by BI" table, with a Gold rating, and help you trust it faster. Definition Propagation Data lineage can help you propagate definitions across your data warehouse by identifying identical data points. From a governance perspective, data lineage proved extremely powerful to propagate metadata. Users can easily propagate a definition across the data systems. If a column is used in another table, without ongoing any transformation than it is fair to assume that in 99% of the cases the definition will be the same. Now, imagine your data catalog is connected to the column-level lineage metadata, you can be 10x more efficient in definition governance by documenting only the sources. If a popular column is used across hundreds of tables without any transformation, you can safely propagate the definition and attributes in a click. Data lineage helps data governance teams from big data or enterprise companies deploy metadata management efforts in a scalable way. Indeed, in an environment with big data resources and numerous databases, understanding how the data flows and its provenance can be challenging. There is a lot of repetitive tasks.‍ Data Privacy Regulation (GDPR and PII mapping) Data lineage can help you tag personal information and track access rights ‍ The logic is really similar to the logic above. Tagging personal or sensitive information can be a real pain. Today, it is a time consuming task. Thanks to the lineage you can understand and track access rights along with dependence.Often you realize, you have some weakness in your roles policy. You have don't have access to a child table or a parent table, but you have access to the middle table. This can be right, but it can also be dangerous as you can reconstruct/trace back the sensitive data. Data lineage is really powerful to improve data governance, access rights management and comply to GDPR regulation.‍ Data assets clean up or technology migration Data lineage can help identify resources you need to power the most valuable data resources ‍Data teams sometimes want to switch technologies. For example, an enterprise company moving from an old on-premise Oracle database to a cloud data warehouse like Snowflake or BigQuery. These big migrations are really expensive as everything is connected to everything. Tracing back dependence is a business critical process before even starting such a project. As part of this process, you want to identify data resources that you can delete versus those you want to migrate. This bird's eye view helps leading architecture thinking. When and why did data lineage become so popular? Data lineage recently gained traction. There are several reasons to that. The first one is that entreprises have more data, sooner than 10 years ago. There's an increased need for metadata management tools. The second is that the data industry structured itself with tools and processes. This makes computing lineage for cloud technologies and companies much more easy and reliable. The last reason is the emergence of harsh data regulations across the continents (GDPR, HIPAA, etc) that makes lineage a must-have. Those three structural changes are compounding trends. They will grow drastically over time. Companies will continue to gather more data and recruit more people to analyse it. The data industry will continue to structure itself and bring standard processes. Compliance to data regulation is today more a problem of entreprise companies but will soon arrive to mid-market and SMBs as tooling will make it easier to assess it. For all these reasons, data lineage is entering the space for good and you might want to start investing in such a tool sooner than later. More modern data stack analysis? Find more benchmarks and analysis on the modern data stack . We write about all the processes involved when leveraging data assets: from the modern data stack to data teams composition, to data governance. Our blog covers the technical and the less technical aspects of creating tangible value from data. If you're a data leader and would like to discuss these topics in more depth, join the we've created for that! here community At Castor, we are building a data documentation tool for the Notion, Figma, Slack generation. Or data-wise for the Fivetran, Looker, Snowflake, DBT aficionados. We designed our catalog to be easy to use, delightful and friendly. Want to check it out? Reach out to us and we'll show you a demo.

Data Lineage is Like Untangling a Ball of Yarn

#1 If your analysis doesn’t have any bias, then look again Problem Definition A bias is an inclination for or against an idea. Most of the time, this is totally unconscious, it takes place mainly when our results are exactly how we expect them to be. We are all human beings, if we have expectations about something, and after digging the data a bit, our first results are as per our expectations, then we tend to stop right there. When our results aren’t how we expect them to be we can keep digging until there are. How to avoid that? Think about what could make your analysis results wrong. I see two main drivers of such bias. The scope of your analysis Try Changing the date range focus or even the data used may get you different results. The classic challenges deal with seasonality and mix effects. Be mindful of cohorts effects The methodology of your analysis This one flirts with statistics 101, now that you’ve got the right scope of time and data points, think carefully about how you aggregate them to get results. Outliers are to be considered, aggregation metric too. Always check the Mean Versus Median. #2 Most first drafts can be done in Excel That title is a bit provocative. Yes, python is powerful and allows you to save and repeat your data processing. But there is the cost to that. First, it takes time, especially if you’re not a python hotshot. Second, collaboration is tougher with non-tech users. If you need non-code-savvy people to work with you on your data app, then python will slow them down. As a data player, you’ll want to do projects in Python, simply to ramp up. But choose them carefully. If you have a super tight schedule and excel does the job then go for excel. You can migrate later to python as it is always easier to learn one thing at a time. It’s hard to do a brand new data app with a language you’re not comfortable with. First do the analysis with a tool you know well, then migrate it to the new language. #3 Get yourself a tool that keeps your query history Ever got a data request similar to the one you had 3 months ago? Happens too many times per year, wishing you had a nice history of all queries you ran in the last 365 days… Check out to do so, a tool built by me and my team. Castor #4 Don’t fix the data, fix the process that creates it Let’s start with a real-life example. One of the data pipelines of one of my previous companies kept breaking because of a not-unique issue: a table field was supposed to be a primary but there were duplicates. That field was client_id and normally a client was supposed to be in one and only one country. So whenever we had this issue we had to find the client linked to several countries and fix it. We would also remind the sales team of the “one country rule”. Should we make a dedicated alerting system on this specific matter? Should we add a transformation layer on top? Should we remove that “unique” check? None of these. We must (and haven’t yet) simply enforce that rule when the data is created at the source, aka, in Salesforce by Salespeople. As much as possible get to the root cause of your data issues, and make people understand that good data requires processes that are optimized for it. Processes are indeed made first to improve the business, but for the sake of having good data, they must factor in the data dependencies. #5 Share your analysis as widely as possible Too many data players wait for their data app to be perfect before sharing it. Share it now (with a “WIP” disclaimer at the beginning if you want). Do not spend more than a few days without having a peer review of your work. It will give you perspective. Conclusion Yes, hard skills ( , SQL, R…) are key to getting started with your analysis but personally, I am looking more into soft skills (good communication, ability to see the big picture, straight-to-the-point, hacky). Python Happy to have a constructive debate in the comments. Also published on: https://www.castordoc.com/blog/the-5-things-every-data-analyst-should-know

5 Most Important Tips Every Data Analyst Should Know

And Why Governance is the Gordian Knot to all Your Business Problems "Data governance is a measure of a company's control over its data" Data governance is a data management concept. It is a measure of the control a company has over its data. This control can be achieved through high-quality data, visibility on data pipelines, actionable rights management, and clear accountability. Data governance encompasses the people, processes, and tools required to create consistent and proper handling of a company's data. By consistent and proper handling of data, I mean ensure availability, usability, consistency, understandability, data integrity, and data security. The most comprehensive governance model — say, for a global bank — will have a robust data-governance council (often with C-suite leaders involved) to drive it; a high degree of automation with metadata recorded in an enterprise dictionary or data catalog; data lineage traced back to the source for many data elements; and a broader domain scope with ongoing prioritization as enterprise needs shift. Good data governance and privacy model is a mix of people, processes, and software. Data Governance has a Direct Business Impact Data governance isn't just that rusty process that companies have to deploy in order to comply with the regulation. Of course, part of it is a legal obligation, and thank god, but clean governance can have high business outcomes. Here are the main goals of data governance: When Did Data Governance Become a Thing? Timeline and key milestones in the space. ‍ For the past twenty years, the challenge around data has been to build an infrastructure to store and consume data efficiently and at scale. Producing data has become cheaper and easier over the years with the emergence of cloud data warehouses and transformation tools like dbt. Access to data has been democratized thanks to BI tools with BI tools like Looker, Tableau, or Metabase. Now, building nice dashboards is the new norm in Ops and Marketing team. This gave rise to a new problem: decentralized, untrustworthy & irrelevant data and dashboards. Even the most data-driven companies still struggle to get value from data - up to 73% of all enterprise data goes unused. ‍ → 1990-2010: the emergence of the 1st regulation on data privacy In the 1970s, the first data protection regulation in the world was vetted in Hessen, Germany. Since then, data regulation has kept increasing. The 1990's mark the first regulations regarding data privacy with the EU directive on data protection. Yet, compliance with regulation really became a worldwide challenge in the second half of the 2010s with the emergence of GDPR, HIPAA, and other regional regulations on personal data privacy. These first regulations drove data governance for large enterprises. This created an urgency to build tools to handle these new requirements. ‍ →  2010 - 2020: 1st tools to comply with the regulation. C-level realizes data governance becomes a strategic advantage to drive business value. With the increasing complexity of data resources/processes on the one hand and the first fines for GDPR infringement on the other, companies started to build regulatory compliance processes. The 1st pieces of software to organize Governance and Privacy were born with companies like Alation and Collibra. The challenge is simple: enforce traceability across the various data infrastructure in the company. Data governance was then a privilege of enterprise-level companies, the only ones able to afford those tools. On-premise data storage makes it expensive to deploy these software. Indeed, companies like Alation and Collibra had to deploy technology specialists on the field to connect the data to their software. The first version of data governance tools aims at collecting and referencing data resources across the organization's departments. There were several forces at play in this period. It became easier to collect data, cheaper to store it, simpler to analyze it. This led to a Cambrian explosion of the number of data resources. As a result, large companies struggled to have visibility over the work done with data. Data was decentralized, untrustworthy & irrelevant. This chaos brought a new strategic dimension to data governance. More than a compliance obligation, data governance became a key lever to bring about business value. ‍ → 2020+: Towards an automated and actionable data governance With the standardization of the cloud data stack, the paradigm changed. It is easier to connect to the data infrastructure and gather metadata. Where it took 6 months to deploy a data governance tool on a multitude of siloed on-premise data centers in 2012, it can take up to 10 minutes in 2021 on the modern data stack (for example: Snowflake, Looker, and DBT). This gave rise to new challenges: automatization and collaboration. Data governance on excel means maintaining manually 100+ fields, on thousands of tables and dashboards. This is impossible. Data governance with a non-automated tool means maintaining 10+ fields on thousands of tables: this is time-consuming. Doing data governance with a fully automated tool means maintaining 1 or 2 fields only on thousands of tables (literally table and column/field description). For that last part of manual work, you want to leverage the community. Prioritize work based on data consumption (high documentation SLA for popular resources) and democratize usage through a friendly UX. Additionally, you want that data governance tool to be integrated into the rest of the data stack. Define something once and find it everywhere: whether this is a table definition, a tag, a KPI, a dashboard, access rights, or data quality results. ‍ Data Governance Challenges Are Not the Same for Everyone Diverse governance's use-cases based on industry needs and company size ‍ There are two main drivers for data governance programs: Level of regulation needed in the industry Data regulation pushes the minimum bar of data governance processes higher. It requires businesses to add controls, reporting, and documentation. This is a need to ensure transparency over sometimes unclear processes. Level of complexity of the data assets Having strong governance becomes increasingly important with the exponential growth of data resources, tools, and people in a company. The level of complexity increases with the scope of business operations (number of lines of business and geographies covered), the velocity of data creation, or the level of automation (decision-making, processes) based on data. ‍ How Do You Set Up Good Data Governance and Privacy Model? Several bricks are needed to enforce data management. Data Architecture (Storage, Modeling, Visualization) Before even talking about data governance, a company needs the basis: a good infrastructure to begin with. Based on business needs and the company's data maturity, the nature of the data architecture can change a lot. Regarding storage, do you go for on-premise or cloud? Data warehouse or a data lake? Regarding modeling: Spark or DBT? in a data warehouse or BI tool? Real-time or batch? Regarding visualization: do you allow anyone to build dashboards or data teams only? Etc. Search and Discovery The first level of any data governance strategy is making sure relevant people can find the relevant datasets to do their analysis or build their AI model. If you don't implement this step, companies end up with a lot of questions on Slack and useless meetings with the engineering teams. The company ends up with a lot of duplicate tables, analyses, and dashboards. It takes valuable time to engineering resources that are needed to perform the next steps. Metadata and Documentation Once you can efficiently find the data, you need to understand it quickly in order to assess if it is going to be useful. For example, you are looking at a dataset called "active_users_revenue_2021".  There is a column "payment." Is this column in € or $? Has it been refreshed this morning, last week, or last year? Does it contain all the data on active users or just the ones in Europe? If I remove a column, will this break important dashboards for the marketing or finance team? etc. Data Quality Now that you have data stored in scalable infrastructure that everyone can find and understand, you need to trust that what is inside is of high quality. This is why so many data observability and reliability tools were born in the last five years. Data observability is the general concept of using automated monitoring, alerting, and triaging to eliminate data downtime. The two main approaches to data quality are: declarative (manually define thresholds and behavior) or ML-driven (detecting sudden changes in distribution). Security and Access Rights Some data might be more private or strategic than others. Let's say you are a bank; you don't want to give access to the transaction logs to anyone in the company. You need to define access rights, and managing them efficiently can quickly become a struggle as the number and type of people working with data grow. Sometimes, you want to give access to someone for a specific mission and for anything else. What happens when one of your employees is in the finance department but moves to marketing? You need to manage these rights thoroughly and efficiently. Compliance and Regulation This one is self-explanatory. You need to list all assets, report on personal information and usage to comply with regulations. For now, only enterprise companies are targeted by regulators; it is just a question of time before smaller companies start receiving fines. ‍ Where Does Data Governance Fit in the Modern Data Stack? Data governance brings trust from the raw data sources to domain expert dashboards. ‍ The typical data flow is the following : You collect data from various sources from your business. It can be product logs, marketing, and website data, payment and sales logs, etc. You extract that information with tools like Fivetran, Stitch, or Airbyte. You then store this data in a data warehouse (Snowflake, Redshift, Bigquery, Firebolt, to name the most popular). The data warehouse is both a place to store and transform your data to refine it. The new trending transformation layer for the past 3 years is DBT. It enables to perform data transformation in SQL within the data warehouse while implementing software engineering good practices. At last, the transformation helps you build your "data mart," the golden standard in terms of refined data. The visualization brick helps domain experts visualize this gold-level data to share insights throughout the whole organization. These steps are happening on different tools with a high level of abstraction. It is hard to keep a bird's eye view of what happens under the hood. This is what data governance is bringing to the table. You can see how the data flows, where the pipeline breaks, where risks lie, where to put your energy as a data manager, etc. Also published at: . https://www.castordoc.com/blog/what-is-data-governance-and-privacy

Why Businesses Need Data Governance

After 7 years working as Senior Data Manager at Criteo and as Head of Data at Payfit, and after interviewing 200+ data leaders for user research, I am starting to have a good overview of data stacks in action. So, by popular demand, I’m dropping here and there some advice on the trends I observed. Earlier in March, I had two different calls: one from a former colleague, now head of data at , and one from a friend, now head of data at . Both had taken a new job and had to build a complete data stack, from scratch! I assume this could interest others. Here is the piece of advice I gave them. Swile Cajoo Before we start, here is a Decision Matrix that you can use for benchmark purposes: Ease of Use Integration in Cloud environment Community & Documentation Governance Capabilities Pricing To keep things simple, we’ll just follow the data, from source to reporting. Data Warehouse Where you store your data for analytics needs A data warehouse is a place you store and organize your data before analysts use it My recommendation is simple: If your dev stack runs on Google Cloud, then go for BigQuery (BQ) If your dev stack runs on AWS, then go for Snowflake (SF) Here are a few reasons. Redshift was indeed the first big player on the Cloud Data Warehouse market, but it means they’ve made mistakes that followers could avoid, and they haven’t changed much for the last few years. Redshift has a poor UI/UX, especially for end-users (data consumers). BigQuery and Snowflake are quite similar in terms of features and pricing models (based on usage: BQ bills you by scanned data, Snowflake bills you by query time). A slight difference is that the first one is stronger when the whole stack is run on Google Cloud while the second is slightly more “Analytics” focused, making it a perfect match for AWS users. Snowflake is heading towards being a full data platform (multi-cloud, Snowpipe, jobs, data share, python/js functions) with a friendly user interface. BigQuery’s strength is its integration within the Google Cloud systems: read/write from GCS, IAM rights management, billing, Google Spreadsheets native integration, …) Ingestion Get Data in your Data Warehouse Stitch and Fivetran are extracting data from all your business sources and bring them to your Data Warehouse For data originating from a SaaS, use or . These two companies do a great job at maintaining these data dumps for their thousands of clients, and their pricing is linear as well as predictable. The sources they can retrieve data from is numerous, check these and . StitchData Fivetran here here You may want to cherry-pick some tables to ingest yourself, for frequency reasons or whatever. Make sure to leverage the numerous open-source taps made available . here Want a mix between the fully managed and the fully homemade? Check . You’ll have to host it but it comes with the necessary bunch of features, sources, and destinations. Airbyte provides similar features, depending on the plan you’re on. But focuses more on event tracking, it’s a must-have to sync your marketing campaigns with user activity. Think of Segment as the big ears listening to what is happening on your website/app. It also demands an ongoing tagging plan. An alternative is which does not integrate that well with other data sources but does not require a tagging plan Segment Contentsquare Do not worry, there is still plenty of DE work to do: you’ve got ingestion pipelines to build for all your data. If your PROD is in NoSQL (such as Mongo), then you’ve got the transformation to do before ingesting the data in your data warehouse. If your PROD is already in a SQL format, then you can dump all needed data in your analytics warehouse, and directly move on to the next paragraph production Transformation From raw data to model/reports ready Like any raw material, you want to transform it to deliver value. Use DBT to transform data. First things first, (note the L now in the middle). In recent years, storage costs decreased to a point (< $30 per Tb/month) where you can now load all raw data in your data warehouse, and then do the transformation in SQL. Keep in mind that storage in SF and BQ is as cheap as S3 storage is, it’s computing that costs money. ETL is dead, long live ELT Now, putting the spotlight on what became the obvious tool in 90% of the data teams I’ve talked to: the notorious . There is even a new job title that comes with it: . DBT is open source, quite easy to set up, and comes with a bunch of interesting features oriented on quality, such as unit tests. DBT Analytics Engineer The general idea is to load all your data in your warehouse, then use DBT to create your datamart ie: refined data, ready to be used for analytics purposes. As a side note, DBT is not a scheduler, you’ll still have to schedule it. It’s your scheduler who’ll launch the DBT executions… which naturally leads me to the scheduler section below. One last note: end of last year Google , it’s now free and well-integrated with Big Query, but the community is not that big. acquired Dataform Scheduler Run job 1 then run job 2 In any transformation process, there is a schedule. Things have to happen in a certain order. Use Airflow for data transformation scheduling. The market standard for scheduling really boils down to Airflow, the question is which deployment setup you want? You can go for: Self-Managed: dockerized airflow Managed: AWS version / GCP Version Fully-Managed: Astronomer Astronomer helps you get off the ground faster. But there is a cost for that… Data Viz Make your shiny dashboards Once you have a high-value dataset, you will want to show the output in the sexiest and most understandable way. Data visualization with , , or is the best option. Tableau Looker Metabase This one is tough, as competition is fierce within this segment. This will also be the only part your business users will see, out of the whole stack you are building. Strong options are , , or . I know that there are a lot of other options out there, that’s just my opinion Tableau Looker Metabase I’ll make a dedicated article on the matter, but to start, go for Metabase. First, on the data Viz side, it’s simple and straightforward, it takes a few minutes to go from a table to a dashboard. Keep in mind that for a while your dashboards will be simple, as you need to educate the different teams. Oh, and it’s free as long as you host it yourself. Data Discovery Find, understand and use data faster High-value datasets or analyses are useless if no one knows they exist. Make sure people do. Castor is the tool for that. Do you think that data discovery is only for people with too much data or many people? Did you know that after a few data dumps from Stitch or Segment, you’ll already have hundreds of tables? On top of that, I’m quite sure that the first data analyst or engineer you’ll hire will make their own data documentation on a google doc or else. Wouldn’t it be better to have that info accessible to everyone and especially to the second hire? At Castor, we provide just that! is a data documentation tool for the Notion-Figma-Slack generation. Or data-wise, for the Fivetran-Looker-Snowflake-DBT aficionados. We designed our catalog to be . Castor easy to use, automated, delightful, and friendly Want to check it out? and we will show you a demo. Reach out to us

How to Build a Data Stack from Scratch

Did You Know Trees Feel the Pull of the Moon? Scientists Link Sap Flow to Lunar Tides

Знаете ли, че дърветата усещат привличането на Луната? Учените свързват соковия поток с лунните приливи и отливи

¿Sabías que los árboles sienten la atracción de la luna? Los científicos relacionan el flujo de savia con las mareas lunares

آیا می دانستید درختان کشش ماه را احساس می کنند؟ دانشمندان جریان شیره را به جزر و مد ماه پیوند می دهند

הידעת שעצים מרגישים את משיכת הירח? מדענים מקשרים בין זרימת מוהל לגאות ירח

木々が月の引力を感じることをご存知ですか？科学者は樹液の流れと月の潮汐を関連付けています

តើអ្នកដឹងទេថាដើមឈើមានអារម្មណ៍ទាញព្រះច័ន្ទ? អ្នកវិទ្យាសាស្ត្រភ្ជាប់លំហូរ Sap ទៅជំនោរតាមច័ន្ទគតិ

Oyebi ete Banzete Eyokaka Kobenda ya Sanza? Bato ya siansi bakangisaka Sap Flow na ba marées lunaires

ເຈົ້າຮູ້ບໍ່ວ່າຕົ້ນໄມ້ຮູ້ສຶກດຶງດວງເດືອນ? ນັກວິທະຍາສາດເຊື່ອມຕໍ່ກະແສ Sap ກັບ The Lunar Tides

ایا تاسو پوهیږئ چې ونې د سپوږمۍ د راښکته کیدو احساس کوي؟ ساینس پوهان د سپو جریان د قمري لندبل سره نښلوي

Waizviziva Here Kuti Miti Inonzwa Kudhonzwa Kwemwedzi? Masayendisiti Anobatanidza Kuyerera Kwemvura kune Lunar Tides

Daraxtlar Oyning tortishishini his qilishini bilarmidingiz? Olimlar dastani oqimini Oy to'lqinlari bilan bog'laydi

你知道树木能感受到月球的引力吗？科学家将树液流与月球潮汐联系起来

Reputation is among humankind's oldest currencies, yet in the digital age, its contours have shifted, expanded, and blurred in fascinating ways. It was in ancient times the force shaping tribal hierarchies, governing trade relations, and even determining survival. But in an increasingly decentralized, nonhierarchical world, reputation transcends simple notions of status and value. It has turned amorphous and multifaceted—a shadow we cast in an interwoven world shaped by perception, data, and context. Let's pause for a moment on this idea of reputation as a shadow. It's not something we control directly, though we might try to influence it. It grows or shrinks depending on the light cast on us—whether it's through public opinion, algorithms, or the ever-watchful eyes of social networks. But isn't it also a reflection of something more profound? Probably a reflection of how others perceive our actions, choices, and associations. But is reputation a reflection of who we really are, or is it just a story others tell themselves about us? The Fragmentation of Reputation in a Decentralized World Historically, reputation was often singular and local. You had a reputation within your village, within your guild, or within your court. It was a rather straightforward concept, invariably coupled with proximity and shared experiences. In contrast, today's decentralized, hyperconnected world fragments reputation into innumerable overlapping circles. Your reputation might be quite different on Twitter than on LinkedIn, than in your local community, or than on a niche forum dedicated to a passion you hold. In this world, reputation is no longer a static asset but a dynamic, ever-changing reflection of the multiplicity of identities we project. This multiplicity creates a curious tension: Can one ever be fully known, or is reputation always partial, incomplete? Each digital space we occupy provides only a sliver of the full story, leaving our reputations vulnerable to misinterpretation, amplification, or distortion. Consider the case of social media influencers. A well-recognized influencer in beauty, for instance, might have an excellent reputation in her community for being honest and creative; outside that circle, they might be dismissed as shallow or materialistic. Similarly, a person held in high regard for their technical expertise in one corner of the Internet could be unknown (or even disliked) in spheres where those skills are not appreciated. Reputation becomes, in a decentralized society, less about the notion of singular truth and more about navigating through a series of shifting truths. Algorithmic Reputation for Me Means The Digital Panopticon As we move into digital platforms, reputation now flows directly from algorithms and data points. Algorithms have become the new gatekeepers that curate and sort individuals based on perceived influence, trustworthiness, and engagement. It could be a credit score or a social network reputation system-invisible hands that mold and shape how others view and treat us, many times without our notice. This mediation through algorithms raises profound questions: Can reputation be trusted when it is distilled into data points?
If the proxy for reputation becomes followers, likes, and endorsers, are we running a serious risk of confusing the appearance of worthiness with integrity, knowledge, or skill? In many ways, the algorithm forms a kind of panopticon, a digital architecture in which we're always in a state of visibility, all performance and all ranking. But in ways quite different from Michel Foucault's panopticon, where a single centric authority could watch over us, today's panopticon is decentralized. We are being judged by everybody and nobody at the same time: peers, algorithms, and anonymous audiences. Reputation Without Hierarchy: Is It Possible? If we step away from reputation as a hierarchical one - where some are "above" and others "below"- we can begin to dream up new possibilities for what it might mean in a more equitable society. In that world, reputation may signal neither rank nor influence but rather signal one's dedication, contribution, or authenticity to a community. That is, it means thinking about reputation in light of shifting the static marker of worth into a dynamic signal of their alignment with the shared values and goals. For example, the open-source software community is sort of a prototype in that respect: the reputation here depends less on financial ability or other forms of power hierarchies. It's based on participation code, support, documentation, and idea development. One acquires reputation not by collecting followers or capital but by adding value to some common endeavor. Even here, in this ostensible flat landscape, reputation plays out through questions of access, influence, and trust. But can this scale outside of small, niche communities? Might we envision a world in which reputation is a function of values and commitments rather than of one's ability to game the system or perform for an audience? The idea of a decentralized, thus-free, manipulation-free reputation system most definitely is one that allured many, yet is highly fraught with risks. How do we prevent new forms of bias, exclusion, or exploitation from emerging? How do we make sure our reputation will stick to meaningful contributions and not superficial impressions? The Future of Reputation: Collective and Fluid We can only imagine that reputation will be even more collective and in flux in the future with the continued rise of decentralized technologies like blockchain. Consider a world in which reputation is decoupled from the performance of an individual but linked to an accumulation of collective experiences, networks, and communities. Your reputation may be a composite of the standing of groups you are part of, projects you work on, or causes you are identifying with. It is a vision that reorients reputation, moving it from the individual and onto the network: It's not about you, but about the company you keep, the ecosystems you nurture, and the values with which you align yourself. It makes reputation less of an achievement badge and more of an interconnection one. Recognizing that, in a decentralized world, none of us stand alone. In the future, reputation could be less something that's a shadow cast by our individual actions and more a constellation: a web of relationships and associations that define who we are within the context of a greater, dynamic system. In that process, this can yield more holistic ways of social capital, wherein the only way reputations are harnessed and maintained is through authentic, sustained participation in things greater than ourselves. Conclusion Once local and unified, reputation has splintered and dispersed itself through the digital world. No longer a badge of merit, it is an unfolding story, context-sensitive, written by algorithmic, communal, and networked agents of evaluation. So, on the way to a more decentralized society, this might also imply that reputation moves from being an individual signifier of standing into a communal, fluid articulation of our positions within larger ecologies. Reputation, at the bottom, is about value, but not necessarily value placed on us by others in some ranking system. It can instead be the value we add to the multiple, intersecting communities we live in and through, and the grace with which we surf the permutations of connection, contribution and trust. As we build this all anew, the challenge will be to make sure that this new kind of reputation is fair and meaningful - one that reflects who we truly are, as opposed to just what we appear to be. Reputation is among humankind's oldest currencies, yet in the digital age, its contours have shifted, expanded, and blurred in fascinating ways. It was in ancient times the force shaping tribal hierarchies, governing trade relations, and even determining survival. But in an increasingly decentralized, nonhierarchical world, reputation transcends simple notions of status and value. It has turned amorphous and multifaceted—a shadow we cast in an interwoven world shaped by perception, data, and context. Let's pause for a moment on this idea of reputation as a shadow. It's not something we control directly, though we might try to influence it. It grows or shrinks depending on the light cast on us—whether it's through public opinion, algorithms, or the ever-watchful eyes of social networks. But isn't it also a reflection of something more profound? Probably a reflection of how others perceive our actions, choices, and associations. But is reputation a reflection of who we really are, or is it just a story others tell themselves about us? The Fragmentation of Reputation in a Decentralized World Historically, reputation was often singular and local. You had a reputation within your village, within your guild, or within your court. It was a rather straightforward concept, invariably coupled with proximity and shared experiences. In contrast, today's decentralized, hyperconnected world fragments reputation into innumerable overlapping circles. Your reputation might be quite different on Twitter than on LinkedIn, than in your local community, or than on a niche forum dedicated to a passion you hold. In this world, reputation is no longer a static asset but a dynamic, ever-changing reflection of the multiplicity of identities we project. This multiplicity creates a curious tension: Can one ever be fully known, or is reputation always partial, incomplete? Can one ever be fully known, or is reputation always partial, incomplete? Each digital space we occupy provides only a sliver of the full story, leaving our reputations vulnerable to misinterpretation, amplification, or distortion. Consider the case of social media influencers. A well-recognized influencer in beauty, for instance, might have an excellent reputation in her community for being honest and creative; outside that circle, they might be dismissed as shallow or materialistic. Similarly, a person held in high regard for their technical expertise in one corner of the Internet could be unknown (or even disliked) in spheres where those skills are not appreciated. Reputation becomes, in a decentralized society, less about the notion of singular truth and more about navigating through a series of shifting truths. Algorithmic Reputation for Me Means The Digital Panopticon As we move into digital platforms, reputation now flows directly from algorithms and data points. Algorithms have become the new gatekeepers that curate and sort individuals based on perceived influence, trustworthiness, and engagement. It could be a credit score or a social network reputation system-invisible hands that mold and shape how others view and treat us, many times without our notice. This mediation through algorithms raises profound questions: Can reputation be trusted when it is distilled into data points? If the proxy for reputation becomes followers, likes, and endorsers, are we running a serious risk of confusing the appearance of worthiness with integrity, knowledge, or skill? Can reputation be trusted when it is distilled into data points? If the proxy for reputation becomes followers, likes, and endorsers, are we running a serious risk of confusing the appearance of worthiness with integrity, knowledge, or skill? In many ways, the algorithm forms a kind of panopticon, a digital architecture in which we're always in a state of visibility, all performance and all ranking. But in ways quite different from Michel Foucault's panopticon, where a single centric authority could watch over us, today's panopticon is decentralized. We are being judged by everybody and nobody at the same time: peers, algorithms, and anonymous audiences. Reputation Without Hierarchy: Is It Possible? If we step away from reputation as a hierarchical one - where some are "above" and others "below"- we can begin to dream up new possibilities for what it might mean in a more equitable society. In that world, reputation may signal neither rank nor influence but rather signal one's dedication, contribution, or authenticity to a community. That is, it means thinking about reputation in light of shifting the static marker of worth into a dynamic signal of their alignment with the shared values and goals. For example, the open-source software community is sort of a prototype in that respect: the reputation here depends less on financial ability or other forms of power hierarchies. It's based on participation code, support, documentation, and idea development. One acquires reputation not by collecting followers or capital but by adding value to some common endeavor. Even here, in this ostensible flat landscape, reputation plays out through questions of access, influence, and trust. But can this scale outside of small, niche communities? Might we envision a world in which reputation is a function of values and commitments rather than of one's ability to game the system or perform for an audience? The idea of a decentralized, thus-free, manipulation-free reputation system most definitely is one that allured many, yet is highly fraught with risks. How do we prevent new forms of bias, exclusion, or exploitation from emerging? How do we make sure our reputation will stick to meaningful contributions and not superficial impressions? The Future of Reputation: Collective and Fluid We can only imagine that reputation will be even more collective and in flux in the future with the continued rise of decentralized technologies like blockchain. Consider a world in which reputation is decoupled from the performance of an individual but linked to an accumulation of collective experiences, networks, and communities. Your reputation may be a composite of the standing of groups you are part of, projects you work on, or causes you are identifying with. It is a vision that reorients reputation, moving it from the individual and onto the network: It's not about you, but about the company you keep, the ecosystems you nurture, and the values with which you align yourself. It makes reputation less of an achievement badge and more of an interconnection one. Recognizing that, in a decentralized world, none of us stand alone. In the future, reputation could be less something that's a shadow cast by our individual actions and more a constellation: a web of relationships and associations that define who we are within the context of a greater, dynamic system. In that process, this can yield more holistic ways of social capital, wherein the only way reputations are harnessed and maintained is through authentic, sustained participation in things greater than ourselves. Conclusion Once local and unified, reputation has splintered and dispersed itself through the digital world. No longer a badge of merit, it is an unfolding story, context-sensitive, written by algorithmic, communal, and networked agents of evaluation. So, on the way to a more decentralized society, this might also imply that reputation moves from being an individual signifier of standing into a communal, fluid articulation of our positions within larger ecologies. Reputation, at the bottom, is about value, but not necessarily value placed on us by others in some ranking system. It can instead be the value we add to the multiple, intersecting communities we live in and through, and the grace with which we surf the permutations of connection, contribution and trust. As we build this all anew, the challenge will be to make sure that this new kind of reputation is fair and meaningful - one that reflects who we truly are, as opposed to just what we appear to be.

Reputation: The Shadow We Cast in a Networked World

السمعة: الظل الذي نلقيه في عالم الشبكات

Reputasiya: Şəbəkələşmiş Dünyada Tutduğumuz Kölgə

Reputació: L'ombra que projectem en un món en xarxa

Reputación: La sombra que proyectamos en un mundo interconectado

評判: ネットワーク化された世界で私たちが投げかける影

평판: 네트워크화된 세상에서 우리가 드리운 그림자

Botumo: Moriti wo re o Lahlelago Lefaseng leo le nago le Neteweke

شهرت: هغه سیوري چې موږ یې په شبکه شوې نړۍ کې اچولي

Reputação: A sombra que projetamos em um mundo em rede

Mukurumbira: Mumvuri Watinokanda Munyika Yakabatana

Sumcad: Hadhka Aan Ku Shubnay Dunida Isku Xidhka Ah

Danh tiếng: Cái bóng của chúng ta trong thế giới mạng

Authors:
(1) Xueguang Ma, David R. Cheriton School of Computer Science, University of Waterloo;
(2) Liang Wang, Microsoft Research;
(3) Nan Yang, Microsoft Research;
(4) Furu Wei, Microsoft Research;
(5) Jimmy Lin, David R. Cheriton School of Computer Science, University of Waterloo. Table of Links Abstract and Introduction Method Experiments Ablation Study and Analysis Related Work Conclusion, Acknowledgements and References Abstract The effectiveness of multi-stage text retrieval has been solidly demonstrated since before the era of pre-trained language models. However, most existing studies utilize models that predate recent advances in large language models (LLMs). This study seeks to explore potential improvements that state-of-the-art LLMs can bring. We conduct a comprehensive study, fine-tuning the latest LLaMA model both as a dense retriever (RepLLaMA) and as a pointwise reranker (RankLLaMA) for both passage retrieval and document retrieval using the MS MARCO datasets. Our findings demonstrate that the effectiveness of large language models indeed surpasses that of smaller models. Additionally, since LLMs can inherently handle longer contexts, they can represent entire documents holistically, obviating the need for traditional segmenting and pooling strategies. Furthermore, evaluations on BEIR demonstrate that our RepLLaMA–RankLLaMA pipeline exhibits strong zero-shot effectiveness. Model checkpoints from this study are available on HuggingFace.1 1 Introduction Text retrieval, which entails identifying and ranking the most relevant documents or text snippets in response to a query, is crucial in various opendomain language comprehension tasks (Petroni et al., 2021), including web search (Bajaj et al., 2016), open-domain question answering (Chen et al., 2017), and fact verification (Thorne et al., 2018). Retrieval also plays an important role in enhancing the effectiveness of large language models (LLMs) in a retrieval-augmented generation (RAG) pipeline (Lewis et al., 2020b; Shi et al., 2023). This approach not only mitigates hallucinations but also enables LLMs to access knowledge that is not captured within their parameters (Yang et al., 2023; Jiang et al., 2023). A typical multi-stage text retrieval pipeline consists of a retriever, designed to efficiently locate the top-k relevant texts from a corpus, and a reranker, which further refines the order of the retrieved candidates to improve output quality (Nogueira and Cho, 2019). Both retrievers and rerankers have significantly benefited from the advent of pre-trained language models based on Transformers (Vaswani et al., 2017) such as BERT (Devlin et al., 2019) and T5 (Raffel et al., 2020). These models are trained to encode queries and documents into vector representations for retrieval (Karpukhin et al., 2020; Lin, 2021) or to directly score the relevance between a query and a document for reranking (Nogueira et al., 2019; Zhuang et al., 2023). Recent large language models with billions of parameters, fine-tuned to follow instructions, such as InstructGPT (Ouyang et al., 2022), GPT-4 (OpenAI, 2023), and LLaMA (Touvron et al., 2023a,b), have exhibited extraordinary capabilities in many NLP tasks, surpassing previous smaller pre-trained language models (Zhao et al., 2023). For retrieval, recent methods such as LRL (Ma et al., 2023), RankGPT (Sun et al., 2023), and PRP (Qin et al., 2023) have explored prompting LLMs to perform zero-shot reranking using pairwise or listwise approaches. These methods leverage LLMs by viewing reranking as text generation. However, we see a number of potential issues. First, these methods do not address the entire multistage pipeline, as it is challenging to cast retrieval from a large corpus as a text generation task. Second, they do not leverage labeled data when available. Finally, these rerankers are not efficient because they do not support parallel scoring and are slowed by their multi-pass decoding design. Therefore, we argue that fine-tuning state-ofthe-art large language models to function as retrievers and rerankers can yield better effectiveness than previous smaller models. This approach can also optimally utilize LLMs within multi-stage pipelines. Thus, we are motivated to investigate the following research question: How do state-ofthe-art large language models perform when specifically fine-tuned for multi-stage text retrieval? Our study aims to answer this question by conducting a comprehensive investigation into finetuning the latest LLaMA-2 model (Touvron et al., 2023b), a state-of-the-art, open-source large language model, as both a retriever and a reranker, which we refer to as RepLLaMA and RankLLaMA, respectively. Specifically, we utilize the MS MARCO (Bajaj et al., 2016) and BEIR (Thakur et al., 2021) datasets for our experiments. Our findings suggest that large language models surpass previous smaller models, achieving state-of-the-art effectiveness for both retrieval and reranking through a straightforward training regime and exhibiting strong zero-shot effectiveness. Furthermore, we observe that LLMs, which are inherently pre-trained on longer contexts, demonstrate potential in representing entire documents, thereby eliminating the need for traditional segmenting and pooling strategies for document retrieval. This paper is available on arxiv under CC 4.0 license. 1 https://huggingface.co/castorini Authors: (1) Xueguang Ma, David R. Cheriton School of Computer Science, University of Waterloo; (2) Liang Wang, Microsoft Research; (3) Nan Yang, Microsoft Research; (4) Furu Wei, Microsoft Research; (5) Jimmy Lin, David R. Cheriton School of Computer Science, University of Waterloo. Authors: Authors: (1) Xueguang Ma, David R. Cheriton School of Computer Science, University of Waterloo; (2) Liang Wang, Microsoft Research; (3) Nan Yang, Microsoft Research; (4) Furu Wei, Microsoft Research; (5) Jimmy Lin, David R. Cheriton School of Computer Science, University of Waterloo. Table of Links Abstract and Introduction Abstract and Introduction Method Method Experiments Experiments Ablation Study and Analysis Ablation Study and Analysis Related Work Related Work Conclusion, Acknowledgements and References Conclusion, Acknowledgements and References Abstract The effectiveness of multi-stage text retrieval has been solidly demonstrated since before the era of pre-trained language models. However, most existing studies utilize models that predate recent advances in large language models (LLMs). This study seeks to explore potential improvements that state-of-the-art LLMs can bring. We conduct a comprehensive study, fine-tuning the latest LLaMA model both as a dense retriever (RepLLaMA) and as a pointwise reranker (RankLLaMA) for both passage retrieval and document retrieval using the MS MARCO datasets. Our findings demonstrate that the effectiveness of large language models indeed surpasses that of smaller models. Additionally, since LLMs can inherently handle longer contexts, they can represent entire documents holistically, obviating the need for traditional segmenting and pooling strategies. Furthermore, evaluations on BEIR demonstrate that our RepLLaMA–RankLLaMA pipeline exhibits strong zero-shot effectiveness. Model checkpoints from this study are available on HuggingFace.1 1 Introduction Text retrieval, which entails identifying and ranking the most relevant documents or text snippets in response to a query, is crucial in various opendomain language comprehension tasks (Petroni et al., 2021), including web search (Bajaj et al., 2016), open-domain question answering (Chen et al., 2017), and fact verification (Thorne et al., 2018). Retrieval also plays an important role in enhancing the effectiveness of large language models (LLMs) in a retrieval-augmented generation (RAG) pipeline (Lewis et al., 2020b; Shi et al., 2023). This approach not only mitigates hallucinations but also enables LLMs to access knowledge that is not captured within their parameters (Yang et al., 2023; Jiang et al., 2023). A typical multi-stage text retrieval pipeline consists of a retriever, designed to efficiently locate the top-k relevant texts from a corpus, and a reranker, which further refines the order of the retrieved candidates to improve output quality (Nogueira and Cho, 2019). Both retrievers and rerankers have significantly benefited from the advent of pre-trained language models based on Transformers (Vaswani et al., 2017) such as BERT (Devlin et al., 2019) and T5 (Raffel et al., 2020). These models are trained to encode queries and documents into vector representations for retrieval (Karpukhin et al., 2020; Lin, 2021) or to directly score the relevance between a query and a document for reranking (Nogueira et al., 2019; Zhuang et al., 2023). Recent large language models with billions of parameters, fine-tuned to follow instructions, such as InstructGPT (Ouyang et al., 2022), GPT-4 (OpenAI, 2023), and LLaMA (Touvron et al., 2023a,b), have exhibited extraordinary capabilities in many NLP tasks, surpassing previous smaller pre-trained language models (Zhao et al., 2023). For retrieval, recent methods such as LRL (Ma et al., 2023), RankGPT (Sun et al., 2023), and PRP (Qin et al., 2023) have explored prompting LLMs to perform zero-shot reranking using pairwise or listwise approaches. These methods leverage LLMs by viewing reranking as text generation. However, we see a number of potential issues. First, these methods do not address the entire multistage pipeline, as it is challenging to cast retrieval from a large corpus as a text generation task. Second, they do not leverage labeled data when available. Finally, these rerankers are not efficient because they do not support parallel scoring and are slowed by their multi-pass decoding design. Therefore, we argue that fine-tuning state-ofthe-art large language models to function as retrievers and rerankers can yield better effectiveness than previous smaller models. This approach can also optimally utilize LLMs within multi-stage pipelines. Thus, we are motivated to investigate the following research question: How do state-ofthe-art large language models perform when specifically fine-tuned for multi-stage text retrieval? Our study aims to answer this question by conducting a comprehensive investigation into finetuning the latest LLaMA-2 model (Touvron et al., 2023b), a state-of-the-art, open-source large language model, as both a retriever and a reranker, which we refer to as RepLLaMA and RankLLaMA, respectively. Specifically, we utilize the MS MARCO (Bajaj et al., 2016) and BEIR (Thakur et al., 2021) datasets for our experiments. Our findings suggest that large language models surpass previous smaller models, achieving state-of-the-art effectiveness for both retrieval and reranking through a straightforward training regime and exhibiting strong zero-shot effectiveness. Furthermore, we observe that LLMs, which are inherently pre-trained on longer contexts, demonstrate potential in representing entire documents, thereby eliminating the need for traditional segmenting and pooling strategies for document retrieval. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv 1 https://huggingface.co/castorini

Fine-Tuning LLaMA for Multi-Stage Text Retrieval

মাল্টি-স্টেজ টেক্সট পুনরুদ্ধারের জন্য ফাইন-টিউনিং LLaMA

Feinabstimmung von LLaMA für mehrstufigen Textabruf

Ajuste fino de LLaMA para la recuperación de texto en varias etapas

Affiner LLaMA pour la récupération de texte en plusieurs étapes

बहु-चरणीय पाठ पुनर्प्राप्ति के लिए LLaMA को परिष्कृत करना

多段階テキスト検索のための LLaMA の微調整

다단계 텍스트 검색을 위한 LLaMA 미세 조정

LLaMA de ajuste fino para recuperação de texto em vários estágios

Точная настройка LLaMA для многоэтапного поиска текста

Çok Aşamalı Metin Erişimi için LLaMA'nın İnce Ayarı

Tinh chỉnh LLaMA để truy xuất văn bản nhiều giai đoạn

用于多阶段文本检索的微调 LLaMA

This paper is under CC 4.0 license. available on arxiv Authors: (1) Jinrui Yang, School of Computing & Information Systems, The University of Melbourne (Email: jinruiy@student.unimelb.edu.au); (2) Timothy Baldwin, School of Computing & Information Systems, The University of Melbourne and Mohamed bin Zayed University of Artificial Intelligence, UAE (Email: (tbaldwin,trevor.cohn)@unimelb.edu.au); (3) Trevor Cohn, School of Computing & Information Systems, The University of Melbourne. Table of Links Abstract and Intro Background and Related Work Multi-EuP Experiments and Findings Language Bias Discussion Conclusion, Limitations, Ethics Statement, Acknowledgements, References, and Appendix 6 Conclusion In this paper, we introduce Multi-EuP, a novel dataset for multilingual information retrieval across 24 languages, collected from European Parliament debates. The demographic information provided by the Multi-EuP dataset serves a dual purpose: not only does it contribute to multilingual retrieval tasks, but it also holds significant potential for advancing research in the realm of fairness and bias. This dataset can play a pivotal role in investigating issues of equitable representations and mitigation of biases within document ranking settings. Multi-EuP facilitates diverse information retrieval (IR) scenarios, encompassing one-vs-one, one-vs-many, and many-vs-many settings. We demonstrated the utility of Multi-EuP as a benchmark for evaluating both monolingual and multilingual IR. Our study reveals the presence of language bias in multilingual IR when employing BM25. We further validate the effectiveness of mitigating this bias through the strategic implementation of whitespace as a language tokenizer. We propose to conduct future work in three main areas. First, we intend to expand our investigation of language bias to encompass a broader range of ranking methods, including neural methods such as mDPR (Zhang et al., 2021), mColBERT (Lawrie et al., 2023) and PLAID-X(Santhanam et al., 2022). Second, we will expand the dataset by developing an automated API to retrieve data published by the European Parliament (EP), thereby ensuring realtime synchronization of our dataset. Lastly, our current experiments have explored language bias only, but we plan to further investigate gender bias, age bias, and nationality bias. Limitations The limitations of the Multi-EuP dataset are notable but navigable. Primarily, the temporal coverage of the dataset is confined to the past three years. This temporal constraint arises due to the fact that, preceding 2020, documents released by the EU were predominantly available in mono-lingual versions only. However, a potential remedy lies in the amalgamation of the Europarl (Koehn, 2005) collection, enabling a more comprehensive and holistic MultiEuP dataset. Furthermore, it is worth noting the domain skew of the dataset, in that Multi-EuP inevitably centers on political matters. While this presents challenges, particularly in terms of the intricate nuances of political language, it inherently serves as an excellent foundational stepping stone for delving into the intricacies of multilingual retrieval. We believe, however, that this dataset can serve as a launching pad for broader explorations encompassing crossdomain and open-domain transfer learning scenarios, thus contributing to the broader landscape of language understanding and retrieval. Ethics Statement The dataset contains publicly-available EP data that does not include personal or sensitive information, with the exception of information relating to public officeholders, e.g., the names of the active members of the European Parliament, European Council, or other official administration bodies. The collected data is licensed under the Creative Commons Attribution 4.0 International licence. [8] Acknowledgements This research was funded by Melbourne Research Scholarship and undertaken using the LIEF HPCGPGPU Facility hosted at the University of Melbourne. This facility was established with the assistance of LIEF Grant LE170100200. We would like to thank George Buchanan for providing valuable feedback. References Luiz Henrique Bonifacio, Israel Campiotti, Roberto de Alencar Lotufo, and Rodrigo Frassetto Nogueira. 2021. mMARCO: A multilingual version of MS MARCO passage ranking dataset. CoRR, abs/2108.13897. Ilias Chalkidis, Manos Fergadiotis, and Ion Androutsopoulos. 2021. MultiEURLEX - a multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6974–6996, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. 8 https://eur-lex.europa.eu/cont Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, 8:454–470. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for opendomain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Online. Association for Computational Linguistics. Omar Khattab and Matei Zaharia. 2020. Colbert: Efficient and effective passage search via contextualized late interaction over BERT. CoRR, abs/2004.12832. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of Machine Translation Summit X: Papers, pages 79–86, Phuket, Thailand. Dawn Lawrie, Eugene Yang, Douglas W. Oard, and James Mayfield. 2023. Neural approaches to multilingual information retrieval. arXiv cs.IR 2209.01335. Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, JhengHong Yang, Ronak Pradeep, and Rodrigo Nogueira. 2021. Pyserini: A Python toolkit for reproducible information retrieval research with sparse and dense representations. https://github.com/ castorini/pyserini. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. CoRR, abs/1611.09268. Ella Rabinovich, Raj Nath Patel, Shachar Mirkin, Lucia Specia, and Shuly Wintner. 2017. Personalized machine translation: Preserving original author traits. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 1074–1084, Valencia, Spain. Association for Computational Linguistics. Razieh Rahimi, Azadeh Shakery, and Irwin King. 2015. Multilingual information retrieval in the language modeling framework. Information Retrieval Journal, 18:246–281. Keshav Santhanam, Omar Khattab, Christopher Potts, and Matei Zaharia. 2022. PLAID: An efficient engine for late interaction retrieval. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, CIKM ’22, page 1747–1756, New York, NY, USA. Association for Computing Machinery. Jörg Tiedemann and Santhosh Thottingal. 2020. OPUSMT – building open translation services for the world. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 479–480, Lisboa, Portugal. European Association for Machine Translation. Eva Vanmassenhove and Christian Hardmeier. 2018. Europarl datasets with demographic speaker information. In Proceedings of the 21st Annual Conference of the European Association for Machine Translation, page 391, Alicante, Spain. Denny Vrandeciˇ c and Markus Krötzsch. 2014. ´ Wikidata: A free collaborative knowledge base. Communications of the ACM, 57:78–85. Konrad Wojtasik, Vadim Shishkin, Kacper Wołowiec, Arkadiusz Janz, and Maciej Piasecki. 2023. BEIRPL: Zero shot information retrieval benchmark for the Polish language. arXiv cs.IR 2305.19840. Peilin Yang, Hui Fang, and Jimmy Lin. 2017. Anserini: Enabling the use of lucene for information retrieval research. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1253–1256. Xinyu Zhang, Xueguang Ma, Peng Shi, and Jimmy Lin. 2021. Mr. TyDi: A multi-lingual benchmark for dense retrieval. arXiv cs.CL 2108.08787. A. Appendix [8] https://eur-lex.europa.eu/content/ legal-notice/legal-notice.html

Multi-EuP: Analysis of Bias in Information Retrieval - Conclusion, Limitations, and Ethics Statement

Rethinking Cryptoeconomics - Part 4: Rethinking MEV Dynamics for Equitable Blockchain Societies

Where the agora meets the blockchain. It's not a revolution with blaring trumpets; it's the quiet evolution of governance. In the timeless drama of "Antigone" (442 BCE), King Creon faces the paradox of enforcing city laws at the expense of divine mandates. A tale that underscores the need for collective wisdom in navigating the complexities of governance. Now, imagine blending this ancient wisdom with a sprinkle of cutting-edge tech magic. That's the future I sketch here—one where the agora meets the blockchain, and democracy gets a contemporary makeover. Unveiling a Modern Agora: Athenian Vibes in a Digital Forum This vision begins by reviving the spirit of Athenian direct democracy in a digital agora—a virtual gathering place. Echoing the days when citizens actively shaped their city's fate, our agora is a nod to the ancient forums, now translated into digital conversations where citizens can voice their opinions and make decisions collectively. Building Trust in the Digital Age: The Subtle Magic of Blockchain Now, here's where the tech magic quietly slips in. Behind the scenes is blockchain, a digital ledger that ensures transparency and security without much fuss. Think of it as the quiet custodian of our democratic decisions, recording each move without the drama. It's the backstage hero, keeping our democratic show tamper-proof and trustworthy. Economic Empowerment: Councils for Every Voice Shifting gears to the economic stage, we introduce worker and consumer councils for democratic economic planning—modern assemblies inspired by the Athenian ethos of direct democracy, not the liberal representative “democracies” that function more like oligarchies. These councils give individuals a seat at the economic table, where they can shape production, influence working conditions, and even have a say in product design and distribution. It's democracy at work, quite literally. AI: The Wise Sidekick in Decision-Making This vision doesn't stop at historical echoes; it embraces the future with artificial intelligence (AI). AI algorithms work behind the scenes, offering councils a helping hand in decision-making. It's not about replacing human insights but enhancing them. The councils get a digital sidekick, making informed decisions a breeze. Governance in Layers: Nested Federated Assemblies In the governance realm, we're introducing nested federated assemblies—a multilayered approach inspired by ancient Greek polis. Local autonomy meshes seamlessly with global collaboration. It's a subtle self-reflective dance of decentralized decision-making that ensures decisions are finely tuned to local needs while tackling broader societal challenges, expressing self-awareness, and placing sustainable limitations on economic growth. Equity in Action: A Classless Society in the Making Addressing societal imbalances, this vision champions balanced divisions of labor and fair compensation. Automation steps in to handle the mundane, freeing individuals to pursue more fulfilling roles. It's a quiet revolution, striving to close the wealth and power gap and reimagine societal structures without much fanfare. Education: Nurturing the Informed Citizens of Tomorrow Education becomes the cornerstone, mirroring the essence of Athenian paideia, the city, and broader society. We're fostering critical thinking, technology literacy, and civic responsibility. Citizens graduate not just with degrees but with the tools and experiences to navigate the nuanced world of self-governance and economics. It's education as an invisible superpower. A Decisive Moment for Democracy: A Nudge from Castoriadis As we stand at this crossroads, philosopher Cornelius Castoriadis offers a gentle nudge. It's a call to shift perspectives, a subtle rebellion against the dominance of the economic sphere. Castoriadis invites us to reconnect with our essential mortality, shaping a future where autonomy triumphs over barbarism. A quiet revolution in the making, on the ontological plane. In conclusion, this future society weaves together the wisdom of antiquity with a sprinkle of contemporary tech enchantment. The agora meets the blockchain, and democracy gets a fundamental, yet low-key, makeover. It's not a revolution with blaring trumpets; it's the quiet evolution of governance, where ancient ideals dance with modern possibilities. Welcome to the future—a place where democracy isn't just a system but a subtle symphony of voices shaping our collective destiny. delegat0x is a libertarian anti-capitalist R&D Engineer in the crypto space. They write about the intersections between philosophy, politics, media, alternatives to capitalism, social movements, and collective autonomy. Image produced by delegate0x using DALL-E Also published here

How Democracy Can Be Revitalized With an Ancient Twist and a Tech Upgrade: A Peek Into the Future

Stories, News & Mentions about CASTOR

Categories

Trending Topics

Stories, News & Mentions about CASTOR

Castor’s Stories on HackerNoon

Categories

Trending Topics