Robustness of Sensitive Data Detection Against Indonesian Noisy Text in Shadow AI

This study evaluates the trade-off between deterministic pattern-matching, fine-tuned discriminative, and reasoning paradigms for sensitive data detection across noisy Indonesian text. The evaluation processes five distinct entity types across formal, code-mixed, and slang registers using 1,500 stratified prompts, with statistical validation conducted via paired permutation tests. The reasoning paradigm achieves the highest mean F1-scores for NIK at 0.928 and credit cards at 0.948, with recall degradation of only 0.094, showing strong robustness. The fine-tuned discriminative model leads in bank accounts at 0.949 and emails at 1.000 but proves highly fragile to informal register shifts. Pattern-matching leads in phone numbers at 0.952 but records zero credit card recall. A stark efficiency contrast emerges as pattern-matching reaches 21692.2 tokens per second on local CPU hardware, whereas the discriminative model delivers a 1099.8 speedup over the reasoning paradigm on an NVIDIA A100 GPU architecture, which suffers a heavy mean latency penalty of 13247.369 ms. These structural trade-offs support a layered data loss prevention framework combining deterministic filtering with contextual inference to optimize throughput and accuracy.

Authors:
Grace Esther Simanjuntak, Kuncahyo Setyo Nugroho, Mahmud Isnan, and Bens Pardamean.

The 13th International Conference on ICT for Smart Society (ICISS) 2026

Read Full Text