From Nuisance to News Sense: Pilot Study

Table of Links

• Abstract and Introduction
• Related Work
• The NEWSSENSE Framework
• Pilot Study
• System Overview
• Discussion
• Conclusion, Limitations and Ethics, and References
• A. Appendix: Prompt for Claim Extraction

4 Pilot Study

To gather feedback on the proposed NEWSSENSE interface and provide insights for the actual implementation, we conducted a pilot user study using a NewsReader mockup built with Figma [10]. This section describes the design and results of the study.

4.1 Study Design

We aim to collect feedback on NEWSSENSE’s basic functionality, interface design, and content quality.

The participants were assigned a task of reading a news article using NEWSSENSE and answering a set of questions. The questions focused on the content of the news, how and where the user located information, and their level of trust in the information. These questions aimed to assess the basic functionality of NEWSSENSE in helping readers understand news comprehensively, to motivate further development of the system.

4.2 Results

Following the pilot user study with over 10 users, we identified several key findings. First, all users found NEWSSENSE to be useful in locating important information and verifying the credibility of news articles, aligning with our initial goal. The user-friendly interface of NEWSSENSE was wellreceived, though participants suggested enhancing interactivity to set it apart from other solutions. For instance, displaying real-time feedback like “NewsSense is analyzing the article" during loading.

Regarding content quality, some users found NEWSSENSE limited and suggested increased labeling or categorization within articles. One user noted Two highlighted sentences per page are insufficient for in-depth analysis." User preferences varied for article summarization, with some wanting more key points and others preferring brevity. Contradicting previous feedback, one user preferred “Summarizing key points only, rather than selecting sentences with unclear relevance." Addressing this, NEWSSENSE could allow customization, letting users choose key point count and filter supporting/- contradicting data.

4.3 Study Takeaways

We found that users liked how NewsSense highlighted important sentences from an article. We realized that the claims which are consistent across multiple articles (ie, those which are supported at least once) are likely to be the most important aspects to a given story. NEWSSENSE could inform readers when there are key claims from across the article cluster missing from the article they are reading.

We also found that the bias labels for news venues could be overwhelming, and including them ran counter to our aim of reference-free verification; we eliminated these labels.

Users also appreciated how highlighted sentences functioned as summaries. Consequently we enhance the visibility of text highlights and further emphasize the alignment or contradiction of specific source by making the External Evidence cards colored accordingly.

[10] https://www.figma.com/