Table of Links Abstract and 1. Introduction Abstract and 1. Introduction Prior Work and 2.1 Educational Objectives of Learning Activities
2.2 Multiscale Design
2.3 Assessing Creative Visual Design
2.4 Learning Analytics and Dashboards


Research Artifact/Probe
3.1 Multiscale Design Environment
3.2 Integrating a Design Analytics Dashboard with the Multiscale Design Environment


Methodology and Context
4.1 Course Contexts
4.2 Instructor interviews


Findings
5.1 Gaining Insights and Informing Pedagogical Action
5.2 Support for Exploration, Understanding, and Validation of Analytics
5.3 Using Analytics for Assessment and Feedback
5.4 Analytics as a Potential Source of Self-Reflection for Students


Discussion + Implications: Contextualizing: Analytics to Support Design Education
6.1 Indexicality: Demonstrating Design Analytics by Linking to Instances
6.2 Supporting Assessment and Feedback in Design Courses through Multiscale Design Analytics
6.3 Limitations of Multiscale Design Analytics


Conclusion and References Prior Work and 2.1 Educational Objectives of Learning Activities
2.2 Multiscale Design
2.3 Assessing Creative Visual Design
2.4 Learning Analytics and Dashboards Prior Work and 2.1 Educational Objectives of Learning Activities Prior Work and 2.1 Educational Objectives of Learning Activities 2.2 Multiscale Design 2.2 Multiscale Design 2.3 Assessing Creative Visual Design 2.3 Assessing Creative Visual Design 2.4 Learning Analytics and Dashboards 2.4 Learning Analytics and Dashboards Research Artifact/Probe
3.1 Multiscale Design Environment
3.2 Integrating a Design Analytics Dashboard with the Multiscale Design Environment Research Artifact/Probe Research Artifact/Probe 3.1 Multiscale Design Environment 3.1 Multiscale Design Environment 3.2 Integrating a Design Analytics Dashboard with the Multiscale Design Environment 3.2 Integrating a Design Analytics Dashboard with the Multiscale Design Environment Methodology and Context
4.1 Course Contexts
4.2 Instructor interviews Methodology and Context Methodology and Context 4.1 Course Contexts 4.1 Course Contexts 4.2 Instructor interviews 4.2 Instructor interviews Findings
5.1 Gaining Insights and Informing Pedagogical Action
5.2 Support for Exploration, Understanding, and Validation of Analytics
5.3 Using Analytics for Assessment and Feedback
5.4 Analytics as a Potential Source of Self-Reflection for Students Findings Findings 5.1 Gaining Insights and Informing Pedagogical Action 5.1 Gaining Insights and Informing Pedagogical Action 5.2 Support for Exploration, Understanding, and Validation of Analytics 5.2 Support for Exploration, Understanding, and Validation of Analytics 5.3 Using Analytics for Assessment and Feedback 5.3 Using Analytics for Assessment and Feedback 5.4 Analytics as a Potential Source of Self-Reflection for Students 5.4 Analytics as a Potential Source of Self-Reflection for Students Discussion + Implications: Contextualizing: Analytics to Support Design Education
6.1 Indexicality: Demonstrating Design Analytics by Linking to Instances
6.2 Supporting Assessment and Feedback in Design Courses through Multiscale Design Analytics
6.3 Limitations of Multiscale Design Analytics Discussion + Implications: Contextualizing: Analytics to Support Design Education Discussion + Implications: Contextualizing: Analytics to Support Design Education 6.1 Indexicality: Demonstrating Design Analytics by Linking to Instances 6.1 Indexicality: Demonstrating Design Analytics by Linking to Instances 6.2 Supporting Assessment and Feedback in Design Courses through Multiscale Design Analytics 6.2 Supporting Assessment and Feedback in Design Courses through Multiscale Design Analytics 6.3 Limitations of Multiscale Design Analytics 6.3 Limitations of Multiscale Design Analytics Conclusion and References Conclusion and References Conclusion and References A. Interview Questions A. Interview Questions 6 DISCUSSION & IMPLICATIONS: CONTEXTUALIZING ANALYTICS TO SUPPORT DESIGN EDUCATION According to the model from Suchman’s seminal treatise, Plans and Situated Actions [71], the success of AI hinges on mutual intelligibility between AI and users; in our situated context, the users are instructors and students. This mutual intelligibility depends on how the analytics, which function as linguistic expressions, get interpreted in the situated contexts of their use. More specifically, multiscale design analytics are interpreted in the context of the situated instances of design work that students perform and the pedagogy and assessment that instructors provide. We developed a research artifact to investigate how linking analytics to design instances would affect this interpretative process. We gathered and analyzed qualitative data to find out how instructors experience multiscale analytics when they are contextualized with this indexical linking to the design work they measure. Plans and Situated Actions We first consider RQ2: What specific value can AI-based multiscale design analytics provide to design instructors in situated course contexts? We discuss and derive implications for how indexical presentation techniques, which link analytics to design instances, contextualize analytics and so support their use in abstract and creative tasks, here, in design education. RQ2: What specific value can AI-based multiscale design analytics provide to design instructors in situated course contexts? We then return to RQ1: How, if at all, can AI-based design analytics support instructors’ assessment and feedback experiences in situated course contexts? Here, we discuss and derive implications regarding the particulars of multiscale analytics, what we’ve seen, and their potential to support assessment and feedback in design education. We also consider limitations. RQ1: How, if at all, can AI-based design analytics support instructors’ assessment and feedback experiences in situated course contexts? As Zimmerman et al. articulate, implications are a form of theory produced using a Research through Design approach [85]; according to Gaver, the theory is likely to be “provisional, contingent, and aspirational” [31]. Hence, we intend for investigating whether and how the implications from this study contribute to interfaces for deriving and presenting a range of complex analytics to be a fruitful avenue for future research. Such research can pinpoint, for example, whether particular implications are more useful in certain educational disciplines, in comparison to others. implications 6.1 Indexicality: Demonstrating Design Analytics by Linking to Instances We contribute indexical linking from analytics to visually annotated design instances as a means of demonstrating what they mean. According to Turnbull, indexical statements articulate relationships across contexts to convey new meanings [74]. In the present study, we found the indexicality of the dashboard—i.e., linking analytics with situated design element assemblages—supported instructors in understanding the analytics. A key is the automatic visual annotation of a design to show which scales and clusters were recognized. For example, in I9’s words, “I was able to infer...there is one zoom level that has a particular region...and then they have a different zoom level that focuses on a different region and so on.” Instructors were able to get a quick sense of students’ design organizations and how they were quantified. They were also able to drill down to the work and pinpoint where the analytics mismatch their own interpretations. For one of the designs, as I3 expressed on seeing AI results, “I’m not sure why [it shows here]...a couple [extra] clusters”. “I was able to infer...there is one zoom level that has a particular region...and then they have a different zoom level that focuses on a different region and so on.” Instructors were able to get a quick sense of students’ design organizations and how they were quantified. They were also able to drill down to the work and pinpoint where the analytics mismatch their own interpretations. For one of the designs, as I3 expressed on seeing AI results, “I’m not sure why [it shows here]...a couple [extra] clusters”. Other researchers are admirably pursuing explainable AI strategies for communicating the algorithmic logic of AI to users [e.g., 1, 65]. This research alternatively contributes how linking analytics to instances can visually demonstrate to users what the analytics mean, showing the work of the recognition algorithm in situated contexts of practice, without attending to its underlying logic. Implications. Users, such as instructors and students, are expected to benefit when dashboard interaction directly indexes, that is, presents the linkages between specific analytics and design work instances. Suchman brings attention to how the transparency of AI-based systems—which is based in conveying an AI’s intended purpose to users and establishing accountability—is vital for effectively supporting situated practice [71]. A common AI approach is to get grades for a large set of assignments and build a bottom up recognizer from that data. Such a recognizer is typically based on an arbitrary aggregation of features that can map to an overall grade score, rather than characteristics explicitly discernible to design instructors or students. Alternatively, we crafted the multiscale design analytics of the present research, using contextual, design-based characteristics, to make sense to the design instructors. As a result, these analytics have the potential to inform students reflecting on how to improve their own work, as well as how to comprehend the work of others. Implications Implications Further, our users found value in navigating to specific scales and clusters measured by analytics. In I1’s words, “whether it would be possible to...maybe like pinpoint or just kind of go to the precise scale.” For this, an intermediate representation on the dashboard can prove useful. For example, our findings motivate further investigation, in which beyond presenting one number on the dashboard, users are afforded interaction with a tree visualization [30], indexing the hierarchy of scales and clusters of each design work instance. Such a representation has the potential to further support users in understanding how to use nested structures to convey complex information, going beyond the flatland to utilize a range of scales and clusters. “whether it would be possible to...maybe like pinpoint or just kind of go to the precise scale.” Lastly, in indexing analytics to instances, interfaces will benefit by using animations. Mayer and Moreno showed that adding animations to study material enhances learner understanding [55]. As Tversky explains, animations can aid perception and comprehension of the fine structure of spatial and temporal relationships among different pieces of content [75]. Bederson and Boltman found that animating viewpoint changes in a spatial environment helps users in building a mental map of the information present within the environment [10]. Multiscale design theory extends zoomable user interface theory to focus on how people assemble information elements in order to convey meanings, by using interaction based on space and scale and associated design principles. In our study, we found that using animation helped instructors to understand complex characteristics. In I1’s words, “I now have a better understanding of spatial clusters [with] the animation of colors changing.” We expect animation interactivity features, such as close-ups, zooming, and control of speed [75] to prove useful in supporting navigation to specific scales and clusters. “I now have a better understanding of spatial clusters [with] the animation of colors changing.” Authors:
(1) Ajit Jain, Texas A&M University, USA; Current affiliation: Audigent;
(2) Andruid Kerne, Texas A&M University, USA; Current affiliation: University of Illinois Chicago;
(3) Nic Lupfer, Texas A&M University, USA; Current affiliation: Mapware;
(4) Gabriel Britain, Texas A&M University, USA; Current affiliation: Microsoft;
(5) Aaron Perrine, Texas A&M University, USA;
(6) Yoonsuck Choe, Texas A&M University, USA;
(7) John Keyser, Texas A&M University, USA;
(8) Ruihong Huang, Texas A&M University, USA;
(9) Jinsil Seo, Texas A&M University, USA;
(10) Annie Sungkajun, Illinois State University, USA;
(11) Robert Lightfoot, Texas A&M University, USA;
(12) Timothy McGuire, Texas A&M University, USA. Authors: Authors: (1) Ajit Jain, Texas A&M University, USA; Current affiliation: Audigent; (2) Andruid Kerne, Texas A&M University, USA; Current affiliation: University of Illinois Chicago; (3) Nic Lupfer, Texas A&M University, USA; Current affiliation: Mapware; (4) Gabriel Britain, Texas A&M University, USA; Current affiliation: Microsoft; (5) Aaron Perrine, Texas A&M University, USA; (6) Yoonsuck Choe, Texas A&M University, USA; (7) John Keyser, Texas A&M University, USA; (8) Ruihong Huang, Texas A&M University, USA; (9) Jinsil Seo, Texas A&M University, USA; (10) Annie Sungkajun, Illinois State University, USA; (11) Robert Lightfoot, Texas A&M University, USA; (12) Timothy McGuire, Texas A&M University, USA. This paper is available on arxiv under CC by 4.0 Deed (Attribution 4.0 International) license. This paper is available on arxiv under CC by 4.0 Deed (Attribution 4.0 International) license. available on arxiv available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

How Contextualized AI Analytics Can Strengthen Design Education

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

3D Semantic Instance Maps: Leveraging Foundation Models for Language-Guided Navigation

AI Analysis Appears to Improve How Teachers Evaluate Student Design Projects, Study Finds

AI Tools Take Aim at Assessing Multiscale Design in the Classroom

AI-Powered Analytics Offer New Pathways for Evaluating Multiscale Design Work

The Research Artifact Making AI-Powered Design Insights Visible at Multiple Scales

AI Model Integration Streamlines Feedback in Multiscale Design Environments

3D Semantic Instance Maps: Leveraging Foundation Models for Language-Guided Navigation

AI Analysis Appears to Improve How Teachers Evaluate Student Design Projects, Study Finds

AI Tools Take Aim at Assessing Multiscale Design in the Classroom

AI-Powered Analytics Offer New Pathways for Evaluating Multiscale Design Work

The Research Artifact Making AI-Powered Design Insights Visible at Multiple Scales

AI Model Integration Streamlines Feedback in Multiscale Design Environments

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps