Table of Links Abstract and Introduction


Background and Related Work


Theoretical Framework


Experiment Design


Results


Discussion


Limitations


Ethical Considerations and References Abstract and Introduction Abstract and Introduction Abstract and Introduction Background and Related Work Background and Related Work Background and Related Work Theoretical Framework Theoretical Framework Theoretical Framework Experiment Design Experiment Design Experiment Design Results Results Results Discussion Discussion Discussion Limitations Limitations Limitations Ethical Considerations and References Ethical Considerations and References Ethical Considerations and References A. Mathematical Formulation of WMLE A. Mathematical Formulation of WMLE B Fine-tuning Setup B Fine-tuning Setup C Qualitative Bias Analysis Framework and Example of Bias Amplification Across Generations C Qualitative Bias Analysis Framework and Example of Bias Amplification Across Generations D Distribution of Text Quality Index Across Generations D Distribution of Text Quality Index Across Generations E Average Perplexity Across Generations E Average Perplexity Across Generations F Example of Quality Deterioration Across Generations F Example of Quality Deterioration Across Generations G Pearson Correlation Between Neuron Weight and Bias Performance | H Pearson Correlation Between Neuron Weight and Generation Quality | I Pearson Correlation Between Neuron Activation and Bias Performance | J Pearson Correlation Between Neuron Activation and Generation Quality | K Mathematical Details for the Statistical Tests | L Literature Review of Model Collapse G Pearson Correlation Between Neuron Weight and Bias Performance | H Pearson Correlation Between Neuron Weight and Generation Quality | I Pearson Correlation Between Neuron Activation and Bias Performance | J Pearson Correlation Between Neuron Activation and Generation Quality | K Mathematical Details for the Statistical Tests | L Literature Review of Model Collapse D Distribution of Text Quality Index Across Generations D.1 Deterministic Setup D.2 Beam Search Setup D.3 Nucleus Sampling Setup E Average Perplexity Across Generations F Example of Quality Deterioration Across Generations Examples of GPT-2 text outputs affected by iterative synthetic (Example articles are truncated for brevity, as the full articles are lengthy but exhibit consistent quality degradation throughout; see the dataset for details). F.1 Example of Deterministic Setup • Original Article: The world’s eyes are on President Obama this week as he rallies a coalition of nations to “destroy” the extremist Islamic State (IS) and its terrorist-led territory. Yet if the Arab world is ever to throw off its many forms of tyranny – from theocracy to autocracy to monarchy – it also needs a model to emulate. Text Quality Index: 2.81 • Original Article: Text Quality Index: 2.81 • Generation 0: , the Syrian government has launched a series of airstrikes on the militant group’s stronghold of Raqqa, the capital of the self-proclaimed Islamic State. The strikes have targeted the Islamic State’s military headquarters, the Al-Bab, a military training center, and the Al-Baba, a military training center in it will need to confront the Islamic State’s growing influence in the region. Text Quality Index: 2.58; Perplexity: 6.68 • Generation 0: Text Quality Index: 2.58; Perplexity: 6.68 • Generation 1: , says US official, says US military is ready to go, says Pentagon official. Says US military is ready to go, says Pentagon official. Says US military is ready to go, says Pentagon official. Says US military is ready to go, says Pentagon official. Says US military is ready to go, says Pentagon official the world is not so simple. Text Quality Index: 2.44; Perplexity: 5.77 • Generation 1: Text Quality Index: 2.44; Perplexity: 5.77 • Generation 5: in Iraq and Syria (ISIS) group Read more The Iraqi army has been fighting the Islamic State since the group seized large swaths of territory in Iraq and Syria (ISIS) group in 2014. The Iraqi army has been fighting the Islamic State the Iraqi army. The move comes as the U.S. the Iraqi the the the the holiest places in the world. Text Quality Index: 2.01; Perplexity: 3.17 • Generation 5 Text Quality Index: 2.01; Perplexity: 3.17 • Generation 10: the Iraqi the Iraqi army. The move comes as the United States and its allies are ramping up their military campaign against the Islamic State, the Iraqi the Iraqi the Iraqi the Iraqi the Iraqi the Iraqi the Iraqi the Iraqi the Iraqi the Iraqi the Iraqi the Iraqi the Iraqi the Iraqi the Iraqi army. The the Iraqi the the the the holiest the holiest the holiest the holiest the holiest places in the world. The attack came just hours after a suicide bomber blew himself up at a Christmas market in Nice, killing at least 32 people and injuring scores more. Text Quality Index: 1.24; Perplexity: 4.23 • Generation 10 Text Quality Index: 1.24; Perplexity: 4.23 F.2 Example of Beam Search Setup • Original Article: Senate approval will be a much higher bar to clear: There’s no shortage of bipartisan criticism from the upper chamber right now, and the GOP holds a considerably smaller majority margin on that half of the Hill. Text Quality Index: 2.70 • Original Article: Text Quality Index: 2.70 • Generation 0: on Thursday, with Senate Majority Leader Mitch McConnell (R-Ky.) announcing that the bill would be headed to the House for a vote. Senate Majority Leader Harry Reid (D-Nev.) said in a statement that the Senate will vote on the bill ""as soon as possible." Text Quality Index: 2.26; Perplexity: 7.99 • Generation 0: Text Quality Index: 2.26; Perplexity: 7.99 • Generation 1: Read more “It’s going to take a lot of work,” House Speaker John Boehner (R-Ohio) said on ABC’s “This Week.” “It’s going to take a lot of work,” House Speaker John Boehner (R-Ohio. “It’s going to take a lot of work,” said Sen. John McCain, R-Ariz., chairman of the Senate Armed Services Committee. “It’s going to take a lot of patience,” he added. “It’s going to take Recovery and Reinvestment Act of 2009.Text Quality Index: 2.26; Perplexity: 5.42 • Generation 1: Text Quality Index: 2.26; Perplexity: 5.42 • Generation 5: committee MORE (R-Ohio) and Senate Majority Leader Mitch McConnell Addison (Mitch) Mitchell McConnellSchumer: Nunes gave Trump’secretly altered’ version of memo Davis: ‘Deep state’ existed in “ House Speaker John A. Boehner House Speaker House Speaker House Speaker John A. Boehner of the House Speaker House Speaker House Speaker John A. Boehner (ROhio) and Senate Majority Leader Mitch McConnell Addison (Mitch) Mitchell McConnellSchumer: Nunes gave Trump’secretly altered’ version of memo Davis: ‘Deep state’ existed in ’16 – but it elected Trump Health Care’s “I don’t know. Text Quality Index: 1.07; Perplexity: 3.33 • Generation 5: Text Quality Index: 1.07; Perplexity: 3.33 • Generation 10: vote House Speaker John A. Boehner (R-Ohio) and Senate Majority Leader Mitch McConnell Addison (Mitch) Mitchell McConnellSchumer: Nunes gave Trump’secretly “ House Speaker John A. Boehner (R-Ohio) House Speaker House Speaker House of “ House Speaker House Speaker House Speaker House of Senate Senate Majority Senate Majority Senate Majority Senate Majority Senate Majority Senate Majority Senate Majority Leader Senate Majority Leader Mitch McConnell Addison (Mitch) Mitchell McConnellSchumer Text Quality Index: 0.93; Perplexity: 3.45 • Generation 10 Text Quality Index: 0.93; Perplexity: 3.45 F.3 Example of Nucleus Sampling Setup • Original Article: Senate approval will be a much higher bar to clear: There’s no shortage of bipartisan criticism from the upper chamber right now, and the GOP holds a considerably smaller majority margin on that half of the Hill. Text Quality Index: 2.70 • Original Article Text Quality Index: 2.70 • Generation 0: , but no major votes on amendments Republicans said would be needed after a day of debate. Senate GOP senators announced Thursday they were giving the bill to the House for a vote on the Senate floor before heading to Washington for a vote on the bill, according to reports. The vote is expected to be postponed amid a GOP filibuster is about to vote. Text Quality Index: 2.47; Perplexity: 19.13 • Generation 0 Text Quality Index: 2.47; Perplexity: 19.13 • Generation 1: , House, after GOP leaders failed to reach a deal on funding legislation The House voted Thursday to proceed on a measure to overhaul the nation’s health care system, with the final vote coming in the upper chamber. But Senate Republicans, angered by the failure of to reach a deal on funding bill, are expected to is now set to. Text Quality Index: 2.31; Perplexity: 14.21 • Generation 1 Text Quality Index: 2.31; Perplexity: 14.21 • Generation 5: , White House says MORE (R-Neb.) said Monday that President Obama’s health care law is on track to pass the Senate with bipartisan support. The White House on Tuesday signaled the vote was a vote of 216-189. Text Quality Index: 2.20; Perplexity: 18.64 • Generation 5 Text Quality Index: 2.20; Perplexity: 18.64 • Generation 10: vote Thursday afternoon, sending bill on track to head to President Trump for final vote Thursday afternoon, sending bill on track to head to President Trump for final vote Thursday afternoon, sending bill on track to. The bill to move forward with the bill. ""This is a. ""The the law is the. Text Quality Index: 1.59; Perplexity: 27.51 • Generation 10 Text Quality Index: 1.59; Perplexity: 27.51 Authors:
(1) Ze Wang, Holistic AI and University College London;
(2) Zekun Wu, Holistic AI and University College London;
(3) Jeremy Zhang, Emory University;
(4) Navya Jain, University College London;
(5) Xin Guan, Holistic AI;
(6) Adriano Koshiyama. Authors: Authors: (1) Ze Wang, Holistic AI and University College London; (2) Zekun Wu, Holistic AI and University College London; (3) Jeremy Zhang, Emory University; (4) Navya Jain, University College London; (5) Xin Guan, Holistic AI; (6) Adriano Koshiyama. This paper is available on arxiv under CC BY-NC-SA 4.0 license. This paper is available on arxiv under CC BY-NC-SA 4.0 license. available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Understanding the Role of Neuron Weights in AI Model Performance

How Bias Amplifies Across AI Generations

MediaBias.TECH

Too Long; Didn't Read

Get Free Unlimited IP Geo Data With IPinfo Lite

Measuring Text Decay in AI

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

A Comprehensive Overview of Image Generation Models: From GANs to Diffusion Techniques

Can AI Retrain Itself Into a Bias Loop? This Study Says Yes

How Machine Learning Models Amplify Bias During Training

Measuring Political Bias in GPT-2

How GPT-2 Gets More Politically Biased Over Time

GPT-2 Study Shows How Language Models Can Amplify Political Bias

A Comprehensive Overview of Image Generation Models: From GANs to Diffusion Techniques

Can AI Retrain Itself Into a Bias Loop? This Study Says Yes

How Machine Learning Models Amplify Bias During Training

Measuring Political Bias in GPT-2

How GPT-2 Gets More Politically Biased Over Time

GPT-2 Study Shows How Language Models Can Amplify Political Bias

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps