Legislation is the source code of our society, but it’s often written in a way that’s inaccessible to the very people it governs. Bills can be dozens of pages long, filled with dense legalese and cross-references that make them nearly impossible to read casually. As a developer I believe in using technology to make government more transparent and accessible. I looked at the Texas Legislature’s website and saw a challenge. What if I could use data to train an AI to read any bill and write its own summary? This is the story of how I built an end-to-end pipeline to do just that: from scraping the data, to cleaning the mess, to fine-tuning a powerful language model. The first step was to get the URLs for every bill that had passed, along with its full text and official summary. The Basic Toolkit: BeautifulSoup For simple HTML pages and Python's urllib are the perfect tools. I started by writing a script (all_house_texas.py) to navigate the Bills Filed pages, find all the links to individual bill histories, and extract basic information like the author and caption. Part 1:  Scraping a 20th-Century Government Website # From: leg_module_test.py
# A simple function to turn a URL into a BeautifulSoup object
import urllib.request
from bs4 import BeautifulSoup

def make_soup(url):
	thepage = urllib.request.urlopen(url)
	soupdata = BeautifulSoup(thepage, "lxml")
	return soupdata

website ='https://capitol.texas.gov/BillLookup/History.aspx?LegSess=89R&Bill=SB1'
soup = make_soup(website)

# Find the author's name by its specific HTML ID
for data in soup.findAll('td', id = 'cellAuthors'):
	print(data.text) # From: leg_module_test.py
# A simple function to turn a URL into a BeautifulSoup object
import urllib.request
from bs4 import BeautifulSoup

def make_soup(url):
	thepage = urllib.request.urlopen(url)
	soupdata = BeautifulSoup(thepage, "lxml")
	return soupdata

website ='https://capitol.texas.gov/BillLookup/History.aspx?LegSess=89R&Bill=SB1'
soup = make_soup(website)

# Find the author's name by its specific HTML ID
for data in soup.findAll('td', id = 'cellAuthors'):
	print(data.text) This worked well for the initial data, but I quickly hit a wall. Leveling Up with Selenium for JavaScript The most valuable piece of data wasn't a standard link. It was hidden behind a JavaScript onclick event that opened a new popup window. BeautifulSoup can't execute JavaScript, so it couldn't see the link. This is a common challenge in modern web scraping. The solution? Selenium, a tool that automates a real web browser. Using Selenium, my script could load the page, wait for the JavaScript to render, and then interact with the button just like a human would. Selenium My script texas_leg_txt_from_list.py used this approach to finally get the summary URL. # From: texas_leg_txt_from_list.py
# Using Selenium to find an element and extract its JavaScript-based URL

from selenium import webdriver
from selenium.webdriver.common.by import By
import time

# ... (driver setup code) ...
driver.get('https://capitol.texas.gov/BillLookup/Text.aspx?LegSess=75R&Bill=HB1')
time.sleep(1.5) # Wait for the page to load

# Find the summary link by its ID
bill_summary_link = driver.find_element(By.ID, 'lnkBillSumm')

# Get the 'onclick' attribute, which contains the JavaScript function call
onclick_attribute = bill_summary_link.get_attribute('onclick')

# A bit of string manipulation to extract the URL from the JavaScript
start_index = onclick_attribute.find("('") + 2
end_index = onclick_attribute.find("'", start_index)
bill_summary_url = 'https://capitol.texas.gov/BillLookup/' + onclick_attribute[start_index:end_index]

print(f"Successfully extracted summary URL: {bill_summary_url}")

driver.quit() # From: texas_leg_txt_from_list.py
# Using Selenium to find an element and extract its JavaScript-based URL

from selenium import webdriver
from selenium.webdriver.common.by import By
import time

# ... (driver setup code) ...
driver.get('https://capitol.texas.gov/BillLookup/Text.aspx?LegSess=75R&Bill=HB1')
time.sleep(1.5) # Wait for the page to load

# Find the summary link by its ID
bill_summary_link = driver.find_element(By.ID, 'lnkBillSumm')

# Get the 'onclick' attribute, which contains the JavaScript function call
onclick_attribute = bill_summary_link.get_attribute('onclick')

# A bit of string manipulation to extract the URL from the JavaScript
start_index = onclick_attribute.find("('") + 2
end_index = onclick_attribute.find("'", start_index)
bill_summary_url = 'https://capitol.texas.gov/BillLookup/' + onclick_attribute[start_index:end_index]

print(f"Successfully extracted summary URL: {bill_summary_url}")

driver.quit() With this, I could reliably gather my three key pieces of data for thousands of bills: the main page URL, the full enrolled text, and the official summary text. Part 2: The Cleanup Crew - Turning HTML Chaos into Clean Data Web data is messy. The scraped text was littered with extra whitespace, non-breaking space characters (\xa0), and other HTML artifacts. Before I could feed this to a model, it needed a serious cleanup. My start_analysis_0.py script was dedicated to this crucial step. First, I wrote a simple cleaning function using regex to standardize whitespace and remove junk characters. # From: start_analysis_0.py
# A function to clean raw text scraped from the web
import re

def clean_text(text):
    if not isinstance(text, str):
        return "" # Handle cases where data might not be a string
    
    text = re.sub(r'\s+', ' ', text) # Collapse all whitespace into single spaces
    text = text.replace('\xa0', ' ') # Remove non-breaking spaces
    text = text.replace('__','')
    # ... other replacements ...
    return text.strip() # From: start_analysis_0.py
# A function to clean raw text scraped from the web
import re

def clean_text(text):
    if not isinstance(text, str):
        return "" # Handle cases where data might not be a string
    
    text = re.sub(r'\s+', ' ', text) # Collapse all whitespace into single spaces
    text = text.replace('\xa0', ' ') # Remove non-breaking spaces
    text = text.replace('__','')
    # ... other replacements ...
    return text.strip() Next, I implemented a critical quality control check. A good summary should be significantly shorter than the original text, but not so short that it's useless. I decided to only keep pairs where the summary's length was between 10% and 100% of the full bill's length. This filtered out bad scrapes and irrelevant data. After processing thousands of bills, I plotted a histogram of this ratio. This visualization confirmed that most of my data fell into a healthy distribution, validating my filtering logic. The final, cleaned dataset was saved as a single JSON file, ready for the main event. Part 3: Fine-Tuning a T5 Model Now for the fun part. I chose to use the t5-small model, a powerful and versatile Text-to-Text Transfer Transformer. T5 is great for instruction following tasks. By prepending the text with summarize, I could teach the model that its job was to output a summary. My training pipeline was built in start_analysis_1.py and start_analysis_2.py using the Hugging Face transformers and TensorFlow libraries. t5-small model, teach the model The core process looks like this: Tokenization:  This step converts the text into numerical IDs that the model can understand. I created a pre-processing function to handle this for both the bill text (the input) and the summary (the target label). Tokenization:  This step converts the text into numerical IDs that the model can understand. I created a pre-processing function to handle this for both the bill text (the input) and the summary (the target label). Tokenization: # From: start_analysis_1.py
# This function tokenizes the text and prepares it for the model
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('t5-small')
prefix = "summarize: "

def preprocess_function(examples):
    # Prepare the input text with the "summarize: " prefix
    inputs = [prefix + doc for doc in examples["source_text"]]
    model_inputs = tokenizer(inputs, max_length=512, truncation=True)

    # Tokenize the target summaries
    with tokenizer.as_target_tokenizer():
        labels = tokenizer(examples["target_text"], max_length=128, truncation=True)

    model_inputs["labels"] = labels["input_ids"]
    return model_inputs # From: start_analysis_1.py
# This function tokenizes the text and prepares it for the model
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('t5-small')
prefix = "summarize: "

def preprocess_function(examples):
    # Prepare the input text with the "summarize: " prefix
    inputs = [prefix + doc for doc in examples["source_text"]]
    model_inputs = tokenizer(inputs, max_length=512, truncation=True)

    # Tokenize the target summaries
    with tokenizer.as_target_tokenizer():
        labels = tokenizer(examples["target_text"], max_length=128, truncation=True)

    model_inputs["labels"] = labels["input_ids"]
    return model_inputs Data Loading: I loaded the cleaned JSON into a Hugging Face Dataset object and applied the tokenization.
Training: Finally, I configured the model, set up an optimizer, and kicked off the training process with model.fit(). This is where the magic happens. The model iterates through the data, making predictions, comparing them to the official summaries, and adjusting its internal weights to get better and better. Data Loading: I loaded the cleaned JSON into a Hugging Face Dataset object and applied the tokenization. Data Loading Training: Finally, I configured the model, set up an optimizer, and kicked off the training process with model.fit(). This is where the magic happens. The model iterates through the data, making predictions, comparing them to the official summaries, and adjusting its internal weights to get better and better. Training # From: start_analysis_1.py
# The final training call in TensorFlow
import tensorflow as tf
from transformers import TFAutoModelForSeq2SeqLM, AdamWeightDecay

# ... (dataset and data_collator setup) ...

optimizer = AdamWeightDecay(learning_rate=2e-5, weight_decay_rate=0.01)
model = TFAutoModelForSeq2SeqLM.from_pretrained("t5-small")
model.compile(optimizer=optimizer)

# Let's train!
model.fit(
    x=tf_train_set, 
    validation_data=tf_test_set, 
    epochs=3
)

# Save the fine-tuned model for later use
model.save_pretrained("./t5_small_texas_bills") # From: start_analysis_1.py
# The final training call in TensorFlow
import tensorflow as tf
from transformers import TFAutoModelForSeq2SeqLM, AdamWeightDecay

# ... (dataset and data_collator setup) ...

optimizer = AdamWeightDecay(learning_rate=2e-5, weight_decay_rate=0.01)
model = TFAutoModelForSeq2SeqLM.from_pretrained("t5-small")
model.compile(optimizer=optimizer)

# Let's train!
model.fit(
    x=tf_train_set, 
    validation_data=tf_test_set, 
    epochs=3
)

# Save the fine-tuned model for later use
model.save_pretrained("./t5_small_texas_bills") Part 4: The Verdict - Did It Work? After hours of training, the moment of truth arrived. I fed the model a bill it had never seen before to see what it would produce. Here's an example: **Original Bill Text (Snippet): \ "...AN ACT relating to providing for a reduction of the limitation on the total amount of ad valorem taxes that may be imposed by a school district on the residence homesteads of the elderly or disabled to reflect any reduction in the school district's tax rate and protecting a school district against any resulting loss in local revenue..." **Original Bill Text (Snippet): \ "...AN ACT relating to providing for a reduction of the limitation on the total amount of ad valorem taxes that may be imposed by a school district on the residence homesteads of the elderly or disabled to reflect any reduction in the school district's tax rate and protecting a school district against any resulting loss in local revenue..." **Official Summary (Ground Truth): \ "This Act reduces the limitation on school district ad valorem taxes for elderly or disabled homeowners to reflect tax rate reductions. It ensures that school districts are compensated for any resulting revenue loss. The change applies starting with the 2007 tax year, contingent on voter approval of a related constitutional amendment." **Official Summary (Ground Truth): \ "This Act reduces the limitation on school district ad valorem taxes for elderly or disabled homeowners to reflect tax rate reductions. It ensures that school districts are compensated for any resulting revenue loss. The change applies starting with the 2007 tax year, contingent on voter approval of a related constitutional amendment." **AI-Generated Summary: \ "The bill relates to a reduction of the limitation on the total amount of ad valorem taxes that may be imposed by a school district on the residence homesteads of the elderly or disabled. The bill would also protect a school district against any resulting loss in local revenue." **AI-Generated Summary: \ "The bill relates to a reduction of the limitation on the total amount of ad valorem taxes that may be imposed by a school district on the residence homesteads of the elderly or disabled. The bill would also protect a school district against any resulting loss in local revenue." Analysis: The model correctly identified the main subjects (ad valorem taxes, school districts, elderly/disabled homesteads) and the core action (reducing the tax limitation). While not as eloquent as the human summary, it's accurate and captures the essence of the bill perfectly. It is a functional and useful TL;DR. Analysis: The model correctly identified the main subjects (ad valorem taxes, school districts, elderly/disabled homesteads) and the core action (reducing the tax limitation). While not as eloquent as the human summary, it's accurate and captures the essence of the bill perfectly. It is a functional and useful TL;DR. Analysis: Conclusion This project was a powerful reminder that the most impactful AI applications often rely on a foundation of gritty data engineering. Building the model was the final 20% of the work; the first 80% was the painstaking process of acquiring and cleaning the data. The final model isn't perfect, but it's a powerful proof-of-concept. It demonstrates that we can use modern AI to make complex civic information more accessible to everyone. What's next? What's next? Deploying as an API: Turn this into a public service where anyone can submit a bill's text and get a summary.
Creating a Bot: To post summaries of newly filed bills.
Expanding to other states: Every legislature has the same problem. This entire pipeline could be adapted. Deploying as an API: Turn this into a public service where anyone can submit a bill's text and get a summary. Deploying as an API: Creating a Bot: To post summaries of newly filed bills. Creating a Bot: Expanding to other states: Every legislature has the same problem. This entire pipeline could be adapted. Expanding to other states: If you’re a developer, I encourage you to look at your own local government's data. You might find a similar opportunity to build something that not only sharpens your skills but also serves your community.

This story contains new, firsthand information uncovered by the writer.

This story contains AI-generated text. The author has used AI either for research, to generate outlines, or write the text itself. 

I Built a Python Script to Make 10,000 Laws Understandable

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

AI Agents to Discover Drugs

10 Free Resources to Become a Health Data Scientist

10 Best Python Machine Learning Tutorials

10 Best Reddit Datasets for NLP and Other ML Projects

10 Best Hugging Face Datasets for Building NLP Models

AI Agents to Discover Drugs

10 Free Resources to Become a Health Data Scientist

10 Best Python Machine Learning Tutorials

10 Best Reddit Datasets for NLP and Other ML Projects

10 Best Hugging Face Datasets for Building NLP Models

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps