Uncle Sam’s Data Dash: U.S. Spy Agencies Seek Cost-Efficient Ways to Buy Your Information

US spy agencies are looking for cheaper and quicker ways to access and buy your data from private companies, according to a request for information sponsored by the intelligence community.

When the Office of the Director of National Intelligence (ODNI) declassified its “Report on Commercially Available Information” last year, the public got a taste of how Uncle Sam was buying data from commercial vendors that it would otherwise need a warrant to acquire.

The Intelligence Community seeks “to leverage a data co-op model to reduce the aggregate cost of Commercially Available Information (CAI) and Publicly Available Information (PAI) and accelerate and improve the IC’s acquisition of and access to CAI and PAI“

ODNI, ICDC Request for Information, August 2024

Now, the US spy community has published a request for information (RFI) that would accelerate and expand upon its controversial data purchasing practices to make buying your data even more efficient and cheaper than ever before.

Through “Intelligence Community (IC) Data Co-op (ICDC),” intelligence agencies are looking “to leverage a data co-op model to reduce the aggregate cost of Commercially Available Information (CAI) and Publicly Available Information (PAI) and accelerate and improve the IC’s acquisition of and access to CAI and PAI, while providing quality data management and maintaining adherence to legal and policy requirements for the acquisition and handling of that data,” according to the RFI.

The desire to make data collection cheaper and quicker comes from the fact that spy agencies “are unable to rapidly and efficiently vet and acquire data due to the growing volume of data available for purchase and the increasing number of private sector vendors that sell their own and third-party data.”

Therefore, the intelligence community is looking for “an approach based on a repeatable data management plus civil liberties and privacy best practices to help integrate CAI/PAI data for the entire IC and make it available to the mission users in a more cohesive and effective manner by avoiding duplicative purchases and reducing the overall cost of acquiring, processing (including extracting, loading, transforming, structuring, and cleaning), enriching, and sharing the data.”

They’re basically looking to set up a one-stop shop for streamlining all their spying needs, such as identifying and vetting commercial data vendors, looking after costs, and collecting, structuring, enriching, and sharing data.

“The Sponsor seeks an enterprise approach to integrate information from commercial datasets and publicly available information by establishing a government-funded data co-op, which will serve as a clearinghouse for IC data requirements, a single focal point in negotiations with vendors, a compliance vehicle to ensure the IC’s use of the data complies with owners’ terms and conditions, and a mechanism to pool agency funds to gain access to targeted commercial data sets”

ODNI, ICDC Request for Information, August 2024

According to the RFI, significant challenges include:

• Establishing agile acquisition processes and supporting governance to purchase CAI/PAI for the whole of the IC
• Increasing understanding of the current and evolving CAI/PAI data ecosystems
• Increasing expertise and improving processes to evaluate the data providers, data quality, and cost value of the data being provided
• Shortening acquisition speed while reducing the aggregate cost of acquiring CAI/PAI across the IC
• Protecting intellectual property rights of the suppliers
• Establishing and monitoring a low-side data environment to host and transform raw data at scale using modern techniques to illuminate gaps and provide intelligence while ensuring proper data handling, security and privacy protections of any sensitive data

“The volume and sensitivity of CAI have expanded in recent years mainly due to the advancement of digital technology, including location-tracking and other features of smartphones and other electronic devices, and the advertising-based monetization models that underlie many commercial offerings available on the Internet”

ODNI Declassified Report on Commercially Available Information, January 2022

The Fourth Amendment to the Constitution of the United States of America guarantees protection from illegal search and seizure by the government.

For years, the government has been skirting around the constitution by simply buying information from commercial vendors.

As the declassified ODNI report from last year states, “Commercially Available Information clearly provides intelligence value, whether considered in isolation and/or in combination with other information, and whether reviewed by humans and/or by machines. It also raises significant issues related to privacy and civil liberties.”

“Although CAI may be ‘anonymized,’ it is often possible (using other CAI) to deanonymize and identify individuals, including US persons,” the report adds.

The ICDC request for information was sponsored by the ODNI, the Office of Economic Security and Emerging Technologies (OESET), the Office of the Intelligence Community Chief Data Officer (IC CDO), and the Office of the Open Source Intelligence (OSINT) IC Executive.

“A new Commercially Available Information / Publicly Available Information data acquisition approach is needed for the Intelligence Community to overcome the acquisition and technical limitations of the current model’s rigid and siloed approach”

ODNI, ICDC Request for Information, August 2024