Task-Scoped Evidence Profiles for External Data Catalog Selection and Traceable Machine Learning

Authors

  • Ruobing Wang 1,* https://orcid.org/0009-0006-7883-6248
    • Conceptualization
    • Data Curation
    • Formal Analysis
    • Project Administration
    • Validation
    • Writing – Original Draft Preparation
  • Xingzhi Wang 2 https://orcid.org/0000-0001-8895-0809
    • Formal Analysis
    • Investigation
    • Methodology
    • Supervision
    • Validation
    • Writing – Review & Editing
1 Khoury College of Computer Sciences, Northeastern University, Boston, United States MA 02115
2 College of Engineering - Information Networking Institute, Carnegie Mellon University, Pittsburgh, United States PA 15213
Corresponding author: ruobing2@alumni.cmu.edu

DOI:

https://doi.org/10.63646/ITLF6323

Data Availability Statement

No proprietary dataset is redistributed in this manuscript.

Keywords:

External data catalogs; Data selection; Evidence applicability; Data provenance; Machine learning; Reproducibility

Abstract

External catalogs improve data discovery but do not by themselves establish suitability for a specified machine-learning use. This article defines a task-scoped application profile that connects requirement sets, version-bound evidence, selection decisions and later execution. It distinguishes provider assertions, measurements and AI suggestions, propagates eligible, excluded and deferred outcomes across assets, bundles and run inputs, and separates pre-run obligations from observed fitting behavior. Comparison with metadata, validation and workflow-provenance precedents identifies the contribution as a local decision-and-record contract. A supplied reference implementation includes a machine-readable schema, semantic validator, evidence resolver and provenance export. Executed checks cover 18 controlled fixtures, 19 applicability cases, 31 manifest cases and 12 sequential changes. New retained public snapshots support a 120-month NYC–NOAA retrospective regression and a group-disjoint UCI red-wine classification. Saved inputs and model parameters reproduce the reported predictions. The NYC model does not show consistently better error than a previous-month baseline; wine accuracy is 0.728125 on 320 held-out rows. Neither result estimates a benefit of catalog selection. Historical operational forecasting and future-batch wine use remain deferred. The supplementary package provides source bytes, receipts, code and offline results. Findings support bounded contract conformance and computational reproducibility. Independent requirement adequacy, scientific suitability and comparative workflow benefit remain untested and are addressed through a separate evaluation protocol.

How to Cite

Wang, R., & Wang, X. (2024). Task-Scoped Evidence Profiles for External Data Catalog Selection and Traceable Machine Learning. Journal of AI Analytics and Applications, 2(1), 1-24. https://doi.org/10.63646/ITLF6323

References

Abedjan, Z., Golab, L., & Naumann, F. (2015). Profiling relational data: A survey. The VLDB Journal, 24(4), 557–581. https://doi.org/10.1007/s00778-015-0389-y

Bender, E. M., & Friedman, B. (2018). Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6, 587–604. https://doi.org/10.1162/tacl_a_00041

Breck, E., Polyzotis, N., Roy, S., Whang, S. E., & Zinkevich, M. (2019). Data validation for machine learning. Proceedings of Machine Learning and Systems, 1. https://proceedings.mlsys.org/paper_files/paper/2019/file/928f1160e52192e3e0017fb63ab65391-Paper.pdf

Brickley, D., Burgess, M., & Noy, N. (2019). Google Dataset Search: Building a search engine for datasets in an open Web ecosystem. The World Wide Web Conference, 1365–1375. https://doi.org/10.1145/3308558.3313685

Buneman, P., Khanna, S., & Tan, W.-C. (2001). Why and where: A characterization of data provenance. Database Theory—ICDT 2001, Lecture Notes in Computer Science, 1973, 316–330. https://doi.org/10.1007/3-540-44503-X_20

Castro Fernandez, R., Abedjan, Z., Koko, F., Yuan, G., Madden, S., & Stonebraker, M. (2018). Aurum: A data discovery system. 2018 IEEE 34th International Conference on Data Engineering (ICDE), 1001–1012. https://doi.org/10.1109/ICDE.2018.00094

Chapman, A., Simperl, E., Koesten, L., Konstantinidis, G., Ibáñez, L. D., Kacprzak, E., & Groth, P. (2020). Dataset search: A survey. The VLDB Journal, 29(1), 251-272. https://doi.org/10.1007/s00778-019-00564-x

City of New York. (n.d.). TLC industry indicators [Data set]. NYC Open Data. Retrieved October 7, 2026, from https://data.cityofnewyork.us/d/v6kb-cqej

Cortez, P., Cerdeira, A., Almeida, F., Matos, T., & Reis, J. (2009). Wine Quality [Data set]. UCI Machine Learning Repository. https://doi.org/10.24432/C56S3T

Davidson, S. B., & Freire, J. (2008). Provenance and scientific workflows: Challenges and opportunities. Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, 1345–1350. https://doi.org/10.1145/1376616.1376772

Ehrlinger, L., Schrott, J., Melichar, M., Kirchmayr, N., & Wöß, W. (2021). Data catalogs: A systematic literature review and guidelines to implementation. Database and Expert Systems Applications - DEXA 2021 Workshops, 148-158. https://doi.org/10.1007/978-3-030-87101-7_15

Fisher, R. (1936). Iris [Dataset]. UCI Machine Learning Repository. https://doi.org/10.24432/C56C76

Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., III, & Crawford, K. (2021). Datasheets for datasets. Communications of the ACM, 64(12), 86-92. https://doi.org/10.1145/3458723

Gong, Y., Liu, G., Xue, Y., Li, R., & Meng, L. (2023). A survey on dataset quality in machine learning. Information and Software Technology, 162, 107268. https://doi.org/10.1016/j.infsof.2023.107268

Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., & Fritz, M. (2023). Not what you've signed up for: Compromising real-world LLM-integrated applications with indirect prompt injection. Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security, 79-90. https://doi.org/10.1145/3605764.3623985

Halevy, A., Korn, F., Noy, N. F., Olston, C., Polyzotis, N., Roy, S., & Whang, S. E. (2016). Goods: Organizing Google’s datasets. Proceedings of the 2016 International Conference on Management of Data, 795–806. https://doi.org/10.1145/2882903.2903730

Kapoor, S., & Narayanan, A. (2023). Leakage and the reproducibility crisis in machine-learning-based science. Patterns, 4(9), 100804. https://doi.org/10.1016/j.patter.2023.100804

Koutras, C., Siachamis, G., Ionescu, A., Psarakis, K., Brons, J., Fragkoulis, M., Lofi, C., Bonifati, A., & Katsifodimos, A. (2021). Valentine: Evaluating matching techniques for dataset discovery. 2021 IEEE 37th International Conference on Data Engineering (ICDE), 468-479. https://doi.org/10.1109/ICDE51399.2021.00047

Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems, 33, 9459–9474. https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html

Menne, M. J., Durre, I., Korzeniewski, B., McNeill, S., Thomas, K., Yin, X., Anthony, S., Ray, R., Vose, R. S., Gleason, B. E., & Houston, T. G. (2012). Global Historical Climatology Network - Daily (GHCN-Daily), Version 3.35 [Data set]. NOAA National Climatic Data Center. https://doi.org/10.7289/V5D21VHZ

Menne, M. J., Durre, I., Vose, R. S., Gleason, B. E., & Houston, T. G. (2012). An overview of the Global Historical Climatology Network-Daily database. Journal of Atmospheric and Oceanic Technology, 29(7), 897-910. https://doi.org/10.1175/JTECH-D-11-00103.1

Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019). Model cards for model reporting. Proceedings of the Conference on Fairness, Accountability, and Transparency, 220-229. https://doi.org/10.1145/3287560.3287596

Moreau, L., Clifford, B., Freire, J., Futrelle, J., Gil, Y., Groth, P., Kwasnikowska, N., Miles, S., Missier, P., Myers, J., Plale, B., Simmhan, Y., Stephan, E., & Van den Bussche, J. (2011). The Open Provenance Model core specification (v1.1). Future Generation Computer Systems, 27(6), 743–756. https://doi.org/10.1016/j.future.2010.07.005

National Centers for Environmental Information (NCEI). (n.d.-a). GHCN-Daily readme and technical documentation. Retrieved October 7, 2026, from https://www.ncei.noaa.gov/pub/data/ghcn/daily/readme.txt

National Centers for Environmental Information (NCEI). (n.d.-b). Global Historical Climatology Network daily (GHCNd). Retrieved October 7, 2026, from https://www.ncei.noaa.gov/products/land-based-station/global-historical-climatology-network-daily

New York City Taxi and Limousine Commission (NYC TLC). (n.d.-a). Aggregated reports. Retrieved October 7, 2026, from https://www.nyc.gov/site/tlc/about/aggregated-reports.page

New York City Taxi and Limousine Commission (NYC TLC). (n.d.-b). Monthly indicators data dictionary. Retrieved October 7, 2026, from https://www.nyc.gov/assets/tlc/downloads/pdf/data_dictionary_monthly_indicators.pdf

Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., VanderPlas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., & Duchesnay, E. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12, 2825–2830. https://jmlr.org/papers/v12/pedregosa11a.html

Peng, R. D. (2011). Reproducible research in computational science. Science, 334(6060), 1226–1227. https://doi.org/10.1126/science.1213847

Pushkarna, M., Zaldivar, A., & Kjartansson, O. (2022). Data Cards: Purposeful and transparent dataset documentation for responsible AI. 2022 ACM Conference on Fairness, Accountability, and Transparency, 1776–1826. https://doi.org/10.1145/3531146.3533231

Sambasivan, N., Kapania, S., Highfill, H., Akrong, D., Paritosh, P., & Aroyo, L. M. (2021). “Everyone wants to do the model work, not the data work”: Data cascades in high-stakes AI. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 1–15. https://doi.org/10.1145/3411764.3445518

Sandve, G. K., Nekrutenko, A., Taylor, J., & Hovig, E. (2013). Ten simple rules for reproducible computational research. PLoS Computational Biology, 9(10), e1003285. https://doi.org/10.1371/journal.pcbi.1003285

Scikit-learn developers. (n.d.-a). load_iris (scikit-learn 1.8.0 documentation). Retrieved October 7, 2026, from https://scikit-learn.org/1.8/modules/generated/sklearn.datasets.load_iris.html

Scikit-learn developers. (n.d.-b). Ridge (scikit-learn 1.8.0 documentation). Retrieved October 7, 2026, from https://scikit-learn.org/1.8/modules/generated/sklearn.linear_model.Ridge.html

Stodden, V., Seiler, J., & Ma, Z. (2018). An empirical analysis of journal policy effectiveness for computational reproducibility. Proceedings of the National Academy of Sciences, 115(11), 2584–2589. https://doi.org/10.1073/pnas.1708290115

Vartak, M., Subramanyam, H., Lee, W. E., Viswanathan, S., Husnoo, S., Madden, S., & Zaharia, M. (2016). ModelDB: A system for machine learning model management. Proceedings of the Workshop on Human-In-the-Loop Data Analytics, 1-3. https://doi.org/10.1145/2939502.2939516

Wang, R. Y., & Strong, D. M. (1996). Beyond accuracy: What data quality means to data consumers. Journal of Management Information Systems, 12(4), 5-33. https://doi.org/10.1080/07421222.1996.11518099

Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J. W., da Silva Santos, L. B., Bourne, P. E., Bouwman, J., Brookes, A. J., Clark, T., Crosas, M., Dillo, I., Dumon, O., Edmunds, S., Evelo, C. T., Finkers, R., ... Mons, B. (2016). The FAIR guiding principles for scientific data management and stewardship. Scientific Data, 3(1), 160018. https://doi.org/10.1038/sdata.2016.18

Wilkinson, M. D., Sansone, S. A., Schultes, E., Doorn, P., Bonino da Silva Santos, L. O., & Dumontier, M. (2018). A design framework and exemplar metrics for FAIRness. Scientific Data, 5(1), 180118. https://doi.org/10.1038/sdata.2018.118

World Wide Web Consortium (W3C). (2013). PROV-O: The PROV ontology. W3C Recommendation, 30 April 2013. https://www.w3.org/TR/2013/REC-prov-o-20130430/

World Wide Web Consortium (W3C). (2016). Data on the Web best practices: Data Quality Vocabulary. W3C Working Group Note, 15 December 2016. https://www.w3.org/TR/2016/NOTE-vocab-dqv-20161215/

World Wide Web Consortium (W3C). (2018). ODRL Information Model 2.2. W3C Recommendation, 15 February 2018. https://www.w3.org/TR/2018/REC-odrl-model-20180215/