CSC 380 Publishable Research Project Topics (Fall 2026)

Problem Statement

Software teams increasingly use generative AI to review source code, yet it is unclear how consistently these tools detect real defects, distinguish serious problems from minor concerns, or avoid generating false alarms. An unreliable reviewer could either miss important faults or overwhelm developers with incorrect recommendations.

Objective

Evaluate the accuracy and reliability of one or more LLM-based code reviewers and determine which categories of software defects they detect well, which they miss, and how their performance compares with traditional static-analysis tools or known defect labels.

Scope

Focus on a manageable set of Java, Python, or C/C++ projects. Defects may include logic errors, requirement violations, maintainability issues, security weaknesses, API misuse, and selected code smells. Teams should avoid attempting to evaluate every programming language or every defect class.

Research Method

• Select open-source programs with known bugs or create controlled seeded defects.
• Run one or more LLMs using a consistent review prompt.
• Run one or more established static-analysis or quality tools as baselines.
• Compare reported findings against ground truth.
• Measure precision, recall, F1-score, false positives, false negatives, and performance by defect category.

Anticipated Contribution

The study can provide evidence about when AI-based code review is trustworthy, where human review is still essential, and which defect categories are most or least suitable for automated AI review. The resulting benchmark and evaluation procedure could also be reused in future studies.

Problem Statement

Poor requirements are a major source of software rework. Ambiguous statements, missing constraints, conflicting requirements, unverifiable language, and incomplete acceptance criteria can lead to design errors and costly changes later in development. Generative AI may help detect these defects, but its reliability needs empirical evaluation.

Objective

Determine how effectively generative AI identifies requirement defects and whether structured prompting improves the quality and consistency of its reviews.

Scope

Use a controlled collection of functional and nonfunctional requirements drawn from student projects, public specifications, or instructor-created examples. Limit the defect taxonomy to several clearly defined categories such as ambiguity, incompleteness, inconsistency, non-testability, and missing constraints.

Research Method

• Create or obtain a labeled requirements dataset containing known defects.
• Evaluate one or more AI tools using a simple prompt and a structured requirements-review prompt.
• Compare AI findings with the labeled ground truth or an expert/checklist-based review.
• Measure accuracy, precision, recall, F1-score, and quality of suggested revisions.
• Analyze which defect categories are easiest and hardest for AI to identify.

Anticipated Contribution

The project can produce practical evidence on whether AI should be used as a first-pass requirements reviewer and can identify prompting strategies that improve review quality. The work may also produce a reusable requirements-defect dataset for future CSC 380 research.

Problem Statement

Generative AI can produce unit tests quickly, but high test quantity or code coverage does not necessarily mean high fault-detection ability. Teams need evidence showing whether AI-generated tests are genuinely useful or simply produce redundant or superficial test cases.

Objective

Compare AI-generated unit tests with developer-written tests using mutation testing and conventional coverage metrics to determine which approach detects more faults and at what cost.

Scope

Select approximately 15–30 small or medium-sized open-source modules in one programming language. The study should focus on unit-level testing rather than full-system or UI testing.

Research Method

• Collect existing developer-written unit tests for each selected module.
• Ask an LLM to generate unit tests under a consistent prompt.
• Run both test suites independently.
• Measure statement/branch coverage, mutation score, unique mutants killed, invalid tests, redundant tests, and generation effort.
• Use statistical comparisons across projects where appropriate.

Anticipated Contribution

The study can clarify whether AI-generated tests provide measurable fault-detection value beyond existing human tests, identify conditions under which AI testing is most beneficial, and propose practical guidance for incorporating LLM-generated tests into software quality workflows.

Problem Statement

AI coding assistants can increase development speed, but software that is produced quickly may contain structural weaknesses that are not immediately visible. If AI-generated code is more complex, duplicated, poorly modularized, or difficult to maintain, teams may accumulate technical debt even when the software appears functionally correct.

Objective

Measure and compare maintainability and technical-debt indicators in human-written and AI-generated software created from the same requirements.

Scope

Use a controlled set of small programming tasks or mini-applications. Focus on one language and a manageable number of objective code-quality measures such as complexity, duplication, maintainability index, code smells, documentation, modularity, and static-analysis findings.

Research Method

• Create a common set of programming specifications.
• Obtain human-developed and AI-generated implementations under controlled conditions.
• Verify functional correctness first.
• Evaluate both groups with tools such as SonarQube or language-specific analyzers.
• Compare complexity, duplication, maintainability, code smells, testability, and defect density.
• Optionally ask the AI to refactor its code and test whether metrics improve.

Anticipated Contribution

The study can provide evidence about whether AI-assisted development trades long-term software quality for short-term speed and may produce practical recommendations for code review and refactoring policies when AI coding tools are used.

Problem Statement

AI tools frequently claim to fix bugs, but successful repair depends on the type of defect, available context, and quality of the issue description. A patch that compiles may still be semantically wrong or introduce new failures. Developers need better evidence about the defect classes for which AI-based repair is dependable.

Objective

Measure the success rate of LLM-assisted bug repair across multiple defect categories and identify the project or issue characteristics associated with successful fixes.

Scope

Select a manageable number of GitHub issues or benchmark bugs with known fixes. Focus on several defect categories such as logic errors, API misuse, boundary conditions, configuration errors, null/reference defects, data-processing faults, or concurrency problems.

Research Method

• Collect bugs with reproducible failing tests or known patches.
• Provide the issue description and relevant source context to an LLM.
• Evaluate generated repairs using compilation, existing tests, new regression tests, and semantic comparison.
• Measure successful repair rate, patch size, number of attempts, and success by defect category.
• Compare AI patches with developer patches where appropriate.

Anticipated Contribution

The study can create a defect-category profile showing where AI repair is strong or weak and can provide guidelines for when developers should trust, verify, or avoid AI-generated fixes.

Problem Statement

Code review is essential for quality assurance, but review delays can slow development. Some pull requests receive quick feedback and are merged rapidly, while others remain open for long periods. The factors that influence review efficiency are not always obvious to contributors.

Objective

Identify measurable pull-request characteristics that are associated with review speed, revision effort, and merge outcomes in active open-source projects.

Scope

Mine a carefully selected set of active GitHub repositories. Focus on variables available through repository metadata, such as lines changed, files changed, description length, linked issues, tests, commit count, contributor history, reviewers, and project activity.

Research Method

• Collect pull-request metadata using the GitHub API or archived datasets.
• Define outcomes such as time to first review, time to merge, number of comments, number of revisions, and merged/not merged.
• Clean and normalize the dataset across repositories.
• Use descriptive statistics and regression/classification models to identify influential variables.
• Validate findings across multiple repositories rather than relying on a single project.

Anticipated Contribution

The study can provide evidence-based guidance for writing more reviewable pull requests and may identify repository-level practices that reduce review delay. Because it uses public repository data, it is also well suited to reproducible empirical software engineering.

Problem Statement

Flaky tests sometimes pass and sometimes fail without a relevant source-code change. They reduce developer trust in test suites, waste debugging time, and may delay releases. Because flakiness is often caused by timing, concurrency, shared state, network dependencies, or external services, it can be difficult to identify manually.

Objective

Determine which characteristics of tests and projects are associated with flaky behavior and evaluate whether those characteristics can support a predictive model.

Scope

Use open-source projects with accessible CI histories and repeated test execution data. Limit the study to one CI ecosystem or one programming language if necessary to keep the project feasible.

Research Method

• Identify known or likely flaky tests from CI histories or existing datasets.
• Extract features such as test duration, asynchronous calls, concurrency use, network access, shared state, external dependencies, and test size.
• Compare flaky and non-flaky tests statistically.
• Build a classification model and evaluate precision, recall, F1-score, ROC-AUC, or similar measures.
• Analyze which features contribute most strongly to predictions.

Anticipated Contribution

The project can provide a practical early-warning model for flaky tests and contribute empirical evidence about the software characteristics most strongly associated with nondeterministic testing behavior.

Problem Statement

Software Bills of Materials are intended to identify the components used in an application, but different tools may produce different dependency lists, versions, or vulnerability results. If SBOM outputs are incomplete or inconsistent, organizations may overlook software supply-chain risks.

Objective

Evaluate the completeness, consistency, and vulnerability coverage of several SBOM-generation tools when applied to the same set of open-source applications.

Scope

Select applications from one or two technology ecosystems and compare a small set of established SBOM tools. Focus on dependency discovery, transitive dependencies, version identification, and known-vulnerability mapping.

Research Method

• Select representative open-source applications with known dependency manifests.
• Generate SBOMs using multiple tools.
• Compare discovered components, versions, transitive dependencies, and vulnerability findings.
• Use project manifests or manually verified subsets as ground truth where possible.
• Measure agreement, omissions, false inclusions, and tool-specific strengths and weaknesses.

Anticipated Contribution

The study can provide practical guidance on which SBOM tools are most reliable for particular ecosystems and can document where cross-tool inconsistency may affect software supply-chain assurance.

Problem Statement

Traditional static-analysis tools can detect many syntactic or metric-based problems, but higher-level concepts such as design patterns and some code smells depend on broader context. LLMs may be able to reason about these structures, but their accuracy and consistency remain uncertain.

Objective

Compare the ability of LLMs and traditional static-analysis tools to identify selected design patterns and/or code smells, and determine whether providing architectural or multi-file context improves AI performance.

Scope

Use Java or Python systems with documented patterns or known code smells. Limit the study to several specific pattern/smell categories so that ground truth can be established reliably.

Research Method

• Collect projects with documented design patterns or labeled code smells.
• Run one or more traditional analysis tools.
• Ask an LLM to identify the same patterns/smells under different context conditions.
• Measure precision, recall, F1-score, consistency, and explanation quality.
• Compare isolated-file prompts with multi-file or architectural-context prompts.

Anticipated Contribution

The study can show whether LLMs provide meaningful advantages for context-sensitive software analysis and can identify when traditional tools remain more dependable. It may also produce practical prompting guidance for architecture and maintainability review.

Problem Statement

AI-assisted development may reduce the time needed to write code, but speed alone does not guarantee better software. Developers may accept incorrect suggestions, introduce hidden defects, reduce their own verification effort, or create less maintainable code. A balanced evaluation should measure both productivity and quality.

Objective

Determine whether AI assistance improves overall software-development performance when both completion time and software quality are considered.

Scope

Use a controlled set of small software-engineering tasks. Compare participants or matched conditions with and without AI assistance. Keep the number of tasks and metrics manageable enough to complete within one semester. If human participants are used for publishable research, the team must follow applicable institutional review requirements before data collection.

Research Method

• Design equivalent programming, debugging, or maintenance tasks.
• Assign AI-assisted and non-AI conditions using an approved experimental design.
• Measure completion time, correctness, defects, test coverage, complexity, maintainability, and requirement satisfaction.
• Optionally introduce controlled misleading AI suggestions to evaluate verification behavior.
• Analyze the tradeoff between speed gains and quality outcomes.

Anticipated Contribution

The study can provide a more balanced view of AI-assisted development by showing whether productivity gains are accompanied by equal, better, or worse software quality. The results could inform classroom, team, and organizational policies for responsible AI-assisted software development.