scieee AI-readable full text Open interactive document viewer

Supplementary Material for "Evaluating Foundation Model Integration Strategies for Detecting PII in Java Software Engineering Pipelines"

Falcão, Fabiano Damasceno Sousa; Dias Canedo, Edna

Publisher
Zenodo
DOI
10.5281/zenodo.17993620
Repository
zenodo.org
Language
en

Abstract

ABSTRACT: Context: Java source code may inadvertently embed personally identifiable information (PII), creating privacy, security, and regulatory compliance risks in contemporary software engineering pipelines. In practice, rule based scanners and high recall classifier ensembles can over-generate findings and provide limited support for context dependent interpretation, increasing audit workload and complicating governance in GenAI enabled software supply chains. Objectives: We empirically compare three foundation model based PII detection strategies for Java source code under controlled and reproducible conditions: (i) transformer based classifiers, (ii) a hybrid classifier+LLM pipeline in which an open-weight LLM acts as a contextual judge over classifier generated candidates, and (iii) an LLM-only pipeline that performs end-to-end structured extraction. Methods: We implement the three pipelines on a synthetic, entity annotated Java dataset using shared preprocessing, value level matching, and micro averaged precision, recall, and F1-score. To support auditability and replication, we enforce deterministic settings where supported, strict output sanitization, and comprehensive raw interaction logging. Results: Individual classifier baselines exhibit limited recall in isolation, while OR-ensemble aggregation substantially increases coverage at the cost of higher false positive load. Under the evaluated hybrid configuration and fixed candidate pool, LLM-as-a-judge filtering reduced a portion of false positives but also rejected a non-trivial share of baseline true positives, yielding a recall loss pattern that constrained attainable F1-scores. In contrast, within the same controlled setting, LLM-only extraction achieved a more favorable precision, recall balance when structured output compliance was stable and deterministic sanitization was enforced; effectiveness remained model dependent. Conclusion: The results indicate that recall loss introduced by post-hoc LLM judging is governance relevant, as under reporting sensitive entities can increase compliance risk in software engineering workflows. They further suggest that end-to-end extraction with open-weight foundation models can support privacy aware source code auditing under data governance constraints when embedded in reproducible and auditable pipelines with strict output validation; however, practical reliability depends on structured output robustness and must be assessed per model and configuration.

Full text

Evaluating Foundation Model Integration Strategies for Detecting PII in Java Source Code This Zenodo record provides the complete experimental artifacts supporting an empirical study on detecting Personally Identifiable Information (PII) in Java source code. The package is designed for reproducibility, auditability, and independent verification by reviewers and other researchers. Repository and identifier Zenodo record: 17993620 DOI: 10.5281/zenodo.17993620 When citing these artifacts, please use the DOI above and the metadata provided by the Zenodo record. Archive list 1.dataset-java-pii.zip 2.source-code.zip 3.experiment-configs.zip 4.experiment-results.zip 5.notebooks.zip What this record contains The record is organized as five ZIP archives, each with a focused purpose. Together, they provide everything needed to reproduce the experimental runs and validate the reported results. 1.dataset-java-pii.zip Synthetic Java dataset annotated with PII entities (JSONL + README). 2.source-code.zip Python source code implementing the experimental pipelines (P1, P2, P3) + prompts/schemas. 3.experiment-configs.zip YAML configurations defining the exact parameters for each run (one YAML per run). 4.experiment-results.zip Complete outputs for all runs (detections, metrics, logs, resolved configs). 5.notebooks.zip Google Colab execution notebook used to run the experiments in a GPU-enabled environment. Quick start for reviewers If you only want to verify the numbers reported in the paper, the fastest path is to navigate the results archive and locate the metrics files produced by each run. Open 4.experiment-results.zip and choose the pipeline folder: pipeline-1/, pipeline-2/, or pipeline-3/. Open a run folder (named with the pipeline, model identifier when applicable, and timestamp). Locate the metrics JSON files inside the run folder and compare them with the paper tables/figures. For Pipeline 2 and Pipeline 3, the results archive also contains raw LLM interaction logs that allow auditing what prompts were sent and what raw responses were received. Artifact overview 1) Synthetic dataset (1.dataset-java-pii.zip) The dataset contains 2,200 synthetic Java code samples annotated with PII entities. It is distributed as a single JSONL file (one sample per line) and includes negative samples (no PII) to support false-positive analysis. Key file: data/JavaPII-Synthetic-2200.jsonl Schema highlights: Each sample has a unique id and a Java source code snippet (text). PII entities are listed under fragments with category, value (verbatim), and optional position offsets. Evaluation is designed to support value-level matching by (category, value), with offsets treated as auxiliary metadata. 2) Source code (2.source-code.zip) The source code implements three experimental pipelines: Pipeline 1 (Classifier-only baseline): ensemble of transformer-based token classifiers (high recall candidate extraction). Pipeline 2 (Hybrid): baseline candidates validated by an open-weight LLM acting as a judge, with guardrails and post-processing. Pipeline 3 (LLM-only): end-to-end detection by an LLM with structured prompting, schema validation, deterministic sanitization, and telemetry. The codebase also includes dataset utilities, value-level evaluators, normalization rules, prompt templates (Jinja2), JSON schemas for structured outputs, and detailed logging/telemetry to support auditability. 3) Experiment configurations (3.experiment-configs.zip) Each YAML file corresponds to a single experimental run and specifies the full parameter set required to reproduce that run (dataset paths, model tags, prompts, thresholds, evaluation settings, output layout, and logging). Design principle: experimental behavior is configuration-driven; critical parameters are not hard-coded in the codebase. The configurations were designed to ensure comparability across pipelines and models (same dataset version, consistent evaluation policy, and externally versioned prompts/schemas). 4) Experiment results (4.experiment-results.zip) This archive contains the complete outputs of all executed runs and is intentionally verbose (thousands of files). It includes detection outputs, metrics, resolved configurations, detailed execution logs, and raw LLM interaction logs. Navigation conventions: Top-level folders: pipeline-1/, pipeline-2/, pipeline-3/. Run folders encode pipeline + (model identifier for P2/P3) + timestamp. Each run contains resolved_config.json/yaml to capture the exact configuration as executed. Common artifacts per run include detections*.jsonl, metrics*.json, execution.log, and pipeline-specific audit trails (e.g., prompts rendered for judging, sanitizer drop logs, and schema-validation failures). 5) Execution notebook (5.notebooks.zip) The Google Colab notebook provides a reference execution entrypoint. It installs dependencies, links the extracted artifacts, prepares an inference backend for LLM runs, and executes Pipelines 1–3 using the YAML configurations. The notebook is meant to reproduce the study workflow, not to serve as a tutorial. All experimental parameters remain external and are loaded from configuration files. Reproducibility checklist Use the dataset JSONL file from 1.dataset-java-pii.zip without modification. Run the experiment runners from 2.source-code.zip using the exact YAML files in 3.experimentconfigs.zip. Compare your generated outputs against 4.experiment-results.zip; focus first on metrics JSON files. For LLM-based runs, inspect raw interaction logs to audit prompts and raw model responses. Evaluation policy (high level) Across pipelines, results are evaluated primarily at the value level: Matching is performed on (label/category, value) pairs. Character offsets are optional and may be used for debugging but are not required for matching. Label mapping and equivalence handling follow the executed configuration (recorded in resolved_config.* inside each run folder). Ethical and legal considerations The dataset is synthetic and was created to minimize privacy risks while enabling controlled experimentation. Despite being synthetic, some values may resemble plausible identifiers. Users must not attempt to attribute, link, or infer real identities from dataset content. Users are responsible for compliance with applicable data protection regulations and organizational policies when using, redistributing, or extending these artifacts. License and intended use These artifacts are released for research verification and reproducibility of the associated empirical study. They are not intended as production-ready components without additional validation, security review, and context-specific risk assessment.