Improving Agile Retrospectives through Metacognitive Scaffolding

2025-02-12

An Empirical Study of the Content and Quality of Sprint Retrospectives in Undergraduate Team Software Projects

2024-04-14 · 6 citations

The retrospective, or retro, is a fundamental component of the Agile process, widely used in both software engineering courses and industry. In a retro, teams come together at the end of a sprint to reflect on their team's performance. We conducted an empirical study to explore three research questions concerning retros in undergraduate team projects: (1) What do students reflect on? (2) What is the quality of their reflections? (3) How do teams' retros vary in terms of content and quality? Our study analyzed a corpus of 963 statements documented in the retros of 32 undergraduate software teams (n = 182 students) enrolled in four software engineering courses at two North American universities. A content analysis revealed that teams reflected most often on their work, communication, and collaboration practices. Nearly a third of teams' reflections focused on their general work practices, while nearly half focused on specific areas of the software development lifecycle---most prominently, pull requests, issues, and coding/testing/debugging. An analysis of the quality of teams' retro reflections showed that only 13% provided justification for a strategy to be stopped, continued, or started. An analysis of team-by-team results indicated significant differences in teams' retro content and quality. We compare these results to previous studies of retros in academia and industry, and consider their implications for software engineering education.

Best Practices for Hiring of Teaching Track Faculty Members

2024-12-02

The current hiring landscape for teaching faculty is a perfect storm, where two issues - (1) increasing demand but a lack of trained candidates and (2) highly diverse job expectations - combine to create a job search that can be overwhelming for teaching faculty candidates and disappointing for departments of computing. As departments independently navigate this new reality, they have each come up with their own practices for interviewing and screening teaching faculty candidates. This proliferation of hiring practices has made it increasingly difficult for teaching faculty candidates to identify positions that are a good fit and prepare adequately for the diversity of hiring practices. In response to these challenges, the Computing Research Association - Education committee surveyed faculty across Canada and the USA about their hiring practices. This work culminated in a recent white paper "Best Practices for Hiring Teaching Faculty in Research Computing Departments." Panelists will engage with the recommendations from this recent paper, discuss principles underlying some best practices for the hiring process, and field questions from the audience.

Investigating Reflection in Undergraduate Software Development Teams

2023-03-02 · 5 citations

Metacognition is widely acknowledged as a key soft skill in collaborative software development. The ability to plan, monitor, and reflect on cognitive and team processes is crucial to the efficient and effective functioning of a software team. To explore students' use of reflection--one aspect of metacognition--in undergraduate team software projects, we analyzed the online chat channels of teams participating in agile software development projects in two undergraduate courses that took place exclusively online (n = 23 teams, 117 students, and 4,915 chat messages). Teams' online chats were dominated by discussions of work completed and to be done; just two percent of all chat messages showed evidence of reflection. A follow-up analysis of chat vignettes centered around reflection messages (n = 63) indicates that three-fourths of the those messages were prompted by a course requirement; just 14% arose organically within the context of teams' ongoing project work. Based on our findings, we identify opportunities for computing educators to increase, through pedagogical and technological interventions, teams' use of reflection in team software projects.

Combining GitHub, Chat, and Peer Evaluation Data to Assess Individual Contributions to Team Software Development Projects

ACM Transactions on Computing Education · 2023-05-02 · 13 citations

Assessing team software development projects is notoriously difficult and typically based on subjective metrics. To help make assessments more rigorous, we conducted an empirical study to explore relationships between subjective metrics based on peer and instructor assessments, and objective metrics based on GitHub and chat data. We studied 23 undergraduate software teams ( n = 117 students) from two undergraduate computing courses at two North American research universities. We collected data on teams’ (a) commits and issues from their GitHub code repositories, (b) chat messages from their Slack and Microsoft Teams channels, (c) peer evaluation ratings from the CATME peer evaluation system, and (d) individual assignment grades from the courses. We derived metrics from (a) and (b) to measure both individual team members’ contributions to the team, and the equality of team members’ contributions. We then performed Pearson analyses to identify correlations among the metrics, peer evaluation ratings, and individual grades. We found significant positive correlations between team members’ GitHub contributions, chat contributions, and peer evaluation ratings. In addition, the equality of teams’ GitHub contributions was positively correlated with teams’ average peer evaluation ratings and negatively correlated with the variance in those ratings. However, no such positive correlations were detected between the equality of teams’ chat contributions and their peer evaluation ratings. Our study extends previous research results by providing evidence that (a) team members’ chat contributions, like their GitHub contributions, are positively correlated with their peer evaluation ratings; (b) team members’ chat contributions are positively correlated with their GitHub contributions; and (c) the equality of team’ GitHub contributions is positively correlated with their peer evaluation ratings. These results lend further support to the idea that combining objective and subjective metrics can make the assessment of team software projects more comprehensive and rigorous.

Designing and Assessing Authentic Software Development Projects in Undergraduate Computing Education

2022-08-31 · 1 citations

Team software development projects are a crucial component of undergraduate computing education. Studies of software developers in industry underscore the need for undergraduates to acquire authentic software development experiences in which they contribute to legacy code bases within collaborative software development contexts. Despite their clear educational value, team software development projects are notoriously challenging to implement and assess. Drawing on experience and relevant research results, this Research in Practice Project Activity (RIPPA) brings together computing educators and researchers to discuss, research, and innovate approaches to designing and assessing authentic software development projects. Participants are invited to run empirical studies in their own courses in which they (a) collect subjective and objective measures of team and individual progress in software projects, and (b) explore alternative pedagogical and assessment approaches, including those supported by the use of custom learning management and data analytics tools. By the end of the RIPPA, we aim to publish one or more research papers that advance the state-of-the-art in team software project pedagogy, supporting technology, and assessment based on the results of multi-institutional research studies.

Assessing individual contributions to software engineering projects: a replication study

Computer Science Education · 2022-05-17 · 11 citations

Background and Context Assessing team members’ indivdiual contributions to software development projects poses a key problem for computing instructors. While instructors typically rely on subjective assessments, objective assessments could provide a more robust picture. To explore this possibility, In a 2020 paper, Buffardi presented a correlational analysis of objective metrics and subjective metrics in an advanced software engineering project course (n= 41 students and 10 teams), finding only two significant correlations.Objective To explore the robustness of Buffardi’s findings and gain further insight, we conducted a larger scale replication of the Buffardi study (n = 118 students and 25 teams) in three courses at three institutions.Method We collected the same data as in the Buffardi study and computed the same measures from those data. We replicated Buffardi’s exploratory, correlational and regression analyses of objective and subjective measures.Findings While replicating four of Buffardi’s five significant correlational findings and partially replicating the findings of Buffardi’s regression analyses, our results go beyond those of Buffardi by identifying eight additional significant correlations.Implications In contrast to Buffardi’s study, our larger scale study suggests that subjective and objective measures of individual performance in team software development projects can be fruitfully combined to provide consistent and complementary assessments of individual performance.

The nitrogen biochemistry of vacant lots across an urban land reuse gradient

eCommons (Cornell University) · 2021-01-01

This report was prepared for the New York State Water Resources Institute (WRI) and the Hudson River Estuary program of the New York State Department of Environmental Conservation, with support from the NYS Environmental Protection Fund

Teaching Testing with Modern Technology Stacks in Undergraduate Software Engineering Courses

2021-06-18 · 2 citations

Students' experience with software testing in undergraduate computing courses is often relatively shallow, as compared to the importance of the topic. This experience report describes introducing industrial-strength testing into CMPSC 156, an upper division course in software engineering at UC Santa Barbara. We describe our efforts to modify our software engineering course to introduce rigorous test-coverage requirements into full-stack web development projects, requirements similar to those the authors had experienced in a professional software development setting. We present student feedback on the course and coverage metrics for the projects. We reflect on what about these changes worked (or didn't), and provide suggestions for other instructors that would like to give their students a deeper experience with software testing in their software engineering courses.

Evaluating Commit, Issue and Product Quality in Team Software Development Projects

2021-03-03 · 14 citations

Providing students with authentic software development experiences is essential to preparing them for careers in industry. To that end, many undergraduate courses include a team-based software development experience in which each team works on a different software project. This raises significant challenges for assessing student work and measuring the impact of pedagogical interventions: What do we measure and how, when each team is working on a different project? To address this question, we present a collection of metrics developed using the Goal-Question-Metric framework from the empirical software engineering literature, and an empirical study in which we applied those metrics to assess 23 team software projects involving 94 students at three institutions. Study results suggest that these metrics, which gauge commit, issue, and overall product quality, are sensitive to differences in the quality of teams' processes and products. This work contributes a new metric-based approach to evaluating key aspects of software development processes and products in a wide variety of computing courses.