Search research papers on "flaky test identification", "CI execution evidence", and "source code analysis" using Scispace Deep Search, focusing on empirical studies within the domains of Software Testing, Empirical Software Engineering, and Software Reliability to compare the effectiveness of CI execution evidence against source code analysis alone for flaky-test identification. And I need to find a gap and novelty.This is the topic Beyond Source Code: Evaluating the Contribution of CI Execution Evidence to Flaky-Test Identification Research domain: Software Testing, Empirical Software Engineering and Software Reliability. Core idea: Investigate whether evidence from test execution and continuous-integration (CI) logs helps identify flaky tests more reliably than analysing source code alone.

Research gap and novelty assessment

Your topic is promising, but the broad claim “CI execution evidence has not been used to identify flaky tests” is no longer defensible. In the literature retrieved here, methods already use test-result histories, durations, failure symptoms, and CI artifacts, and very recent work explicitly argues that test code alone is often insufficient. The defensible contribution is a controlled, leakage-resistant comparison of the incremental value of CI evidence over a source-code-only baseline—with realistic labels and deployment-oriented outcomes.

What the empirical literature indicates

• Source-code-only prediction is established, but reported performance depends on evaluation design. Flakify predicts from test source code and reports performance on two datasets, including a project-wise validation procedure [citation_21ryxs]. Peeler likewise presents a static predictor [citation_21ryyf]. However, a 2026 preprint reports that code-only detectors’ apparent performance on some benchmarks was inflated by label shortcuts and evaluation protocols; scores fell to constant-baseline levels under project-disjoint evaluation on its counterfactual dataset [citation_19ewz0]. Treat high cross-validation scores as insufficient evidence of practical generalization.
• Existing CI/history approaches already combine runtime and code-change information. One industrial study used outcome history, test duration, and code-evolution features from existing executions; its best model used test flip rate and code-change measures [citation_1wcp4k]. A CI failure-symptom approach matched new failures to historical flaky symptoms and reported high precision in one industrial system [citation_21ryxv]. These are important comparators, but they do not by themselves establish the general incremental value of CI evidence over a matched source-only model across projects.
• The evidence about adding logs is not uniformly positive. A thesis-level study directly compared models with and without log embeddings and reported that adding its log representation reduced predictive performance [citation_21ryxz]. This makes feature representation and evaluation design central—not merely “include logs.”
• Recent work directly sharpens the gap. A 2026 preprint analyzed 86 flaky end-to-end tests mined from CI logs: test code plus CI logs yielded a cause for 42%, while the other cases required further execution evidence [citation_19ewz0]. This is evidence that more execution information can matter for diagnosis, but it is not a broad, controlled estimate of how much CI evidence improves flaky-test identification over code-only prediction.
• Operational errors matter as much as test-level classification. In a Chromium CI study, high-precision test-focused predictors nevertheless missed many fault-triggering failures; failure-focused methods performed better on the reported MCC measure [citation_21ryxm]. A useful evaluation should therefore measure whether the approach distinguishes a flaky failure from a genuine regression, not only whether a test has ever been flaky.

Defensible gap

Not established by this search: a general claim that no previous study has compared code and execution evidence. The literature includes partial comparisons and a recent, directly relevant preprint. What remains well-motivated—and is not resolved by the abstracts retrieved—is a matched, multi-project, out-of-time and out-of-project assessment of the incremental predictive and operational value of well-defined CI evidence over source code alone, with protection against label leakage and class imbalance.

Proposed research questions

1. RQ1—Incremental effectiveness: Under temporal and project-disjoint evaluation, how much does adding CI execution evidence to source-code features change flaky-test identification performance compared with a source-code-only model?
2. RQ2—Evidence contribution: Which CI evidence groups—outcome history and flip rate, duration, failure messages/stack traces, and execution/environment context—provide reliable incremental value, and for which flakiness categories?
3. RQ3—Practical reliability: Does the combined model improve failure-level triage (flaky failure versus likely regression) at an acceptable false-quarantine rate and computational cost?

Novelty you can credibly claim

Frame the novelty as an empirical evaluation of evidence contribution, not simply a new detector that consumes CI logs. The study would add:

• A controlled comparison using the same projects, test instances, labels, and modeling pipeline for (a) source code only, (b) CI evidence only, and (c) source code plus CI evidence.
• A feature-ablation analysis separating execution history, failure text, and environment/context, addressing why one log-embedding study found no gain while other work reports value from history or failure symptoms [citation_21ryxz] [citation_1wcp4k] [citation_21ryxv].
• Evaluation across both unseen future revisions and unseen projects, using natural prevalence rather than only balanced samples. This responds to documented concerns about shortcut labels, unrealistic distributions, and weak generalization [citation_19ewz0] [citation_19oui7].
• A deployment-oriented endpoint: correctly identifying flaky failures without suppressing genuine fault-triggering failures, an important risk demonstrated in Chromium CI [citation_21ryxm].

Avoid claiming “first-ever” until you have checked the full texts and related work of the closest studies—especially the 2026 CI-log preprint, the industrial history-based predictor, and the log-feature thesis. On the present evidence, novelty is strongest as the rigor and scope of the head-to-head evaluation.

Feasible empirical design

Recommended scope: start with one ecosystem (for example, projects with publicly retrievable GitHub CI histories) and a manageable set of languages/build systems. Expand only if comparable outcome histories and logs can be collected reliably.

1. Construct a time-indexed dataset. For each test and CI run, preserve the code revision, result, duration, failure text/stack trace, and available environment/build metadata. Define a prediction cutoff: only information available before the target failure/run may enter the features. Label flakiness using repeated outcomes on the same revision/configuration where possible, supplemented by documented developer-confirmed fixes or adjudication; keep “flaky test” labels distinct from “this particular failure is flaky.”
2. Compare matched feature sets. Train the same model families and tuning budget on source-only, CI-only, and combined features. Run ablations for history, failure text, and environment/context. Include simple baselines (for example, flip-rate and repeated-run rules) as well as one or more established source-code predictors.
3. Prevent leakage. Use forward-in-time testing and project-disjoint testing, with related revisions or duplicate tests grouped so they cannot cross training/test boundaries. Report any residual project or label-selection constraints.
4. Measure useful outcomes. Report precision-recall AUC (important under low flaky-test prevalence), recall at a prespecified false-positive/false-quarantine rate, precision, recall, MCC, calibration, and analysis/runtime cost. For the failure-level task, explicitly count genuine regressions incorrectly classified as flaky. Compare models with paired uncertainty estimates and report results per project and flakiness category, not just a pooled average.
5. State feasibility limits. Public CI logs can be incomplete, inconsistent across providers, or unavailable for some runs; repeated execution is an imperfect reference standard for rare flakiness. These constraints should be documented and tested with sensitivity analyses rather than hidden by a balanced benchmark.

Candidate title: Beyond Source Code: A Leakage-Resistant, Multi-Project Evaluation of CI Execution Evidence for Flaky-Test Identification.

Bottom line: the research gap is not whether CI evidence can be used—prior work already uses execution histories, logs, and failure symptoms. It is whether specific, realistically available CI evidence adds generalizable and operationally safe value beyond code-only prediction under fair, leakage-resistant comparisons.

Search scope and limits

I could not run Scispace Deep Search from this session. I searched the available academic databases (Semantic Scholar, OpenAlex, and PubMed) using deep semantic retrieval; the result set contained 99 selected records, and the optional lexical concept-planning step failed while semantic retrieval continued. Paper content was restricted to abstracts, so the gap assessment is preliminary and does not establish exhaustive coverage or “first” priority. Several especially relevant records are recent preprints or a thesis; verify their publication status and inspect the closest full texts before finalizing a novelty claim.