John F. Hurdle, MD, PhD
- Nutritional Data Mining
- Clinical Natural Language Processing
- Ethics Committees, Research
- Health Services
- Departments: Biomedical Informatics - Professor
Academic Office Information
421 Wakara Way, Room: Suite 140, Room 2028
Salt Lake City, UT 84108
Main Research Interests: practical Natural Language Processing for clinical and biomedical text applications; finding smart and inexpensive ways to assess the grocery quality of the foods households buy at grocery stores and online.
Education: Dr. John F. Hurdle earned his MD from the University of Colorado (1981) and his MS in Computer Science from Columbia University the same year, followed by a PhD in Computer Science from the University of Utah (1994). He took his informatics postdoctoral training here at Utah (1996-97). Following his postdoctoral training, Dr. Hurdle worked as a research clinical informaticist at the Salt Lake Veterans Medical Center’s (VAMC) Geriatric Research, Education, and Clinical Center (1998-2004). While there, he was the Principal Investigator on two VA Health Services Research and Development (HSRD) Grants, the first such grants funded at that VAMC in over two decades. His small center grant from HSRD has grown into a multi-million dollar HSRD enterprise at the VAMC and includes the VA’s primary informatics research infrastructure, VINCI.
At the University of Utah: He joined the Department of Biomedical Informatics (BMI) as a regular faculty member in 2005. In 2007 he served as a Senior Fellow at the National Library of Medicine. Dr. Hurdle holds the rank of Professor. His work in the BMI spans clinical informatics, clinical research informatics, and public health/consumer health informatics. In addition to a long-standing commitment to improving patient safety, he currently directs a lab that focuses on clinical natural language processing (NLP) and another lab that focuses on nutrition data mining (NDM) . The POET NLP lab builds tools to unlock the content of clinical narratives using NLP, so that this content can be used to assist the healthcare enterprise. The QualMART NDM lab is comprised of University of Utah researchers, graduate students, and post-docs who have expertise in computer science, biomedical informatics, nutrition, and consumer behavior. Our long-term goal focuses on improving the quality of what people eat in order to improve their overall quality of life. Our approach uses weaves of together consumer behavior theory, high-performance computing and simulation, cutting-edge nutrition modeling, and large-scale database management.
Etc.: In addition to research, he is active in both service and education,. He served as the Department’s Director of Graduate Studies from 2008 to 2016 and was the principal investigator of the BMI T15 NLM Training Grant for nearly a decade. Dr. Hurdle has served as a grant reviewer for the National Library of Medicine’s Standing Study Section (2008-2012) and continues to participate in special emphasis panel grant reviews for NLM and other NIH institutes. He served as Chair of the American Medical Informatics Association’s Ethics Committee when it created AMIA’s first code of professional conduct. He also served as member, Vice Chair, and Chair of the University’s Institutional Review Board (1999 – 2016) as well as Chair of the Resource for Genetic and Epidemiologic Research (2011-present), the data governance body overseeing the use of the Utah Population Database.
“I was once asked by a recruiter how I could rationalize staying in academic biomedical informatics rather than work in industry for a lot more money. ‘Oh that’s easy’, I told her, ‘In industry I would be working on what they want – in academics, I work on what I want.’”
I started my informatics career with a keen desire to improve patient safety, especially in the realm of medication errors (which, at the time, were estimated to be causing over 100,000 needless fatalities a year in the US). Over the past 15 years, I developed a very strong interest in applying natural language processing (NLP) tools to clinical narratives. My work on adverse drug events convinced me that there were important signals in clinical notes (aka "unstructured data") that can be used to improve patient care. After completing an NLP Senior Fellowship sponsored by the National Library of Medicine, I was awarded two research grants from the NIH to explore the utility of preprocessing clinical text to make clinical NLP more useful and efficient. My POET natural language processing lab works to develop practical and efficient tools that can extract information locked in clinical narrative text. Increasingly we need the information that can only be found in text to augment traditional structured information like laboratory test results, or medication lists. In combination, these two types of data are essential for modern data analytics and data science apps.
More recently I developed an interest in the nascent field of Nutrition Informatics. So my other lab focuses on nutrition data mining (NDM). Nutrition informatics holds great promise, especially as the United States faces an epidemic of diet-related diseases such as obesity, type II diabetes, and malnutrition. Nutrition informatics is very much in its infancy, and our Department is the first major informatics program to engage nutrition in a serious and principled way. Our primary goal for this work centers on building scalable tools that can measure the food quality of what households buy (in grocery stores or online) coupled to targeted, personalized recommendations designed to nudge households towards a healthier household food environment.
|Research Fellow||National Institutes of Health/NLM
|Senior Research Fellow|
|Postdoctoral Fellowship||University of Utah, Department of Medical Informatics and The Veterans Administration
|Doctoral Training||University of Utah
|Graduate Training||Columbia University
|Professional Medical||University of Colorado, Denver
|University of Maryland (European Division), Computer Studies||Adjunct||Germany|
|University of Maryland, European Division||Director (CIO), Info Systems, Admin Computing, Institutional Research||Germany|
- Brewster PJ, Guenther PM, Jordan KC, Hurdle JF (2017). The Grocery Purchase Quality Index-2016: An innovative approach to assessing grocery food purchases. https://doi.org/10.1016/j.jfca.2017.07.012. J Food Compost Anal.
- Tran LT, Brewster PJ, Chidambaram V, Hurdle JF (2017). An Innovative Method for Monitoring Food Quality and the Healthfulness of Consumers' Grocery Purchases.LID - E457 [pii]LID - 10.3390/nu9050457 [doi]. Nutrients, 9(5).
- Bui DD, Del Fiol G, Hurdle JF, Jonnalagadda S (2016). Extractive text summarization system to aid data extraction from full text in systematic review development. J Biomed Inform, 64, 265-272.
- Doing-Harris KM, Weir CR, Igo S, Shi J, Shao Y, Hurdle JF (2015). POETenceph - Automatic identification of clinical notes indicating encephalopathy using a realist ontology. AMIA Annu Symp Proc, 2015, 512-21.
- Kim Y, Riloff E, Hurdle JF (2015). A Study of Concept Extraction Across Different Types of Clinical Notes. AMIA Annu Symp Proc, 2015, 737-46.
- North JC, Jordan KC, Metos J, Hurdle JF (2015). Nutrition Informatics Applications in Clinical Practice: a Systematic Review. AMIA Annu Symp Proc, 2015, 963-72.
- Tran LT, Brewster PJ, Chidambaram V, Hurdle JF (2015). Towards Measuring the Food Quality of Grocery Purchases: an Estimation Model of the Healthy Eating Index-2010 Using only Food Item Counts. Procedia Food Sci, 4, 148-159.
- Brewster P, Guenther PM, Jordan KC, Hurdle JF (2015). Development and Validation of a Novel Household Grocery Food Purchase Quality Score. FASEB J, 29(1), 131-3.
- He S, Botkin JR, Hurdle JF (2015). An Analysis of Information Technology Adoption by IRBs of Large Academic Medical Centers in the United States. J Empir Res Hum Res Ethics, 10(1), 31-6.
- Tran LT, Brewster PJ, Chidambarab V, Hurdle JF (06/20/2015). Towards Measuring the Food Quality of Grocery Purchases: an Estimation Model of the Healthy Eating Index-2010 Using only Food Item Counts. doi:10.1016/j.profoo.2015.06.020. Procedia Food Sci, 4, 148-159.
- Hurdle JF, Haroldsen SC, Hammer A, Spigle C, Fraser AM, Mineau GP, Courdy SJ (09/15/2014). Identifying clinical/translational research cohorts: ascertainment via querying an integrated multi-source database.IMIA Yearbook of Medical Informatics 2014. Yearbook of Medical Informatics.
- He S, Narus SP, Facelli JC, Lau LM, Botkin JR, Hurdle JF (2014). A domain analysis model for eIRB systems: addressing the weak link in clinical research informatics. J Biomed Inform, 52, 121-9.
- Jones DE, Igo S, Hurdle J, Facelli JC (2014). Automatic extraction of nanoparticle properties using natural language processing: NanoSifter an application to acquire PAMAM dendrimer properties. PLoS One, 9(1), e83932.
- Bradford W, Hurdle JF, LaSalle B, Facelli JC (2014). Development of a HIPAA-compliant environment for translational research data and analytics. J Am Med Inform Assoc, 21(1), 185-9.
- Chidambaram V, Brewster PJ, Jordan KC, Hurdle JF (2013). qDIET: toward an automated, self-sustaining knowledge base to facilitate linking point-of-sale grocery items to nutritional content. AMIA Annu Symp Proc, 2013, 224-33.
- He S, Ganzinger M, Hurdle JF, Knaup P (2013). Proposal for a data publication and citation framework when sharing biomedical research resources. Stud Health Technol Inform, 192, 1201.
- Hurdle JF, Smith KR, Mineau GP (2013). Mining electronic health records: an additional perspective. Nat Rev Genet, 14(1), 75.
- Goodman KW, Adams S, Berner ES, Embi PJ, Hsiung R, Hurdle J, Jones DA, Lehmann CU, Maulden S, Petersen C, Terrazas E, Winkelstein P (2013). AMIA's code of professional and ethical conduct. J Am Med Inform Assoc, 20(1), 141-3.
- Hurdle JF, Haroldsen SC, Hammer A, Spigle C, Fraser AM, Mineau GP, Courdy SJ (2013). Identifying clinical/translational research cohorts: ascertainment via querying an integrated multi-source database. J Am Med Inform Assoc, 20(1), 164-71.
- Workman TE, Fiszman M, Hurdle JF (2012). Text summarization as a decision support aid. BMC Med Inform Decis Mak, 12(1), 41.
- Pestian JP, Matykiewicz P, Linn-Gust M, South B, Uzuner O, Wiebe J, Cohen KB, Hurdle J, Brew C (2012). Sentiment Analysis of Suicide Notes: A Shared Task. Biomed Inform Insights, 5(Suppl 1), 3-16.
- Dorr DA, Cohen AM, Williams MP, Hurdle J (2011). From simply inaccurate to complex and inaccurate: complexity in standards-based quality measures. AMIA Annu Symp Proc, 2011, 331-8.
- Kim Y, Hurdle J, Meystre SM (2011). Using UMLS lexical resources to disambiguate abbreviations in clinical text. AMIA Annu Symp Proc, 2011, 715-22.
- Brinkerhoff KM, Brewster PJ, Clark EB, Jordan KC, Cummins MR, Hurdle JF (2011). Linking supermarket sales data to nutritional information: an informatics feasibility study. AMIA Annu Symp Proc, 2011, 598-606.
- Patterson O, Hurdle JF (2011). Document clustering of clinical narratives: a systematic study of clinical sublanguages. AMIA Annu Symp Proc, 2011, 1099-107.
- Tang H, Poynton MR, Hurdle JF, Baird BC, Koford JK, Goldfarb-Rumyantzev AS (2011). Predicting three-year kidney graft survival in recipients with systemic lupus erythematosus. ASAIO J, 57(4), 300-9.
- Workman TE, Hurdle JF (2011). Dynamic summarization of bibliographic-based data. BMC Med Inform Decis Mak, 11, 6.
- Tang H, Hurdle JF, Poynton M, Hunter C, Tu M, Baird BC, Krikov S, Goldfarb-Rumyantzev AS (2011). Validating prediction models of kidney transplant outcome using single center data. ASAIO J, 57(3), 206-12.
- He S, Hurdle JF, Botkin JR, Narus SP (2010). Integrating a Federated Healthcare Data Query Platform With Electronic IRB Information Systems. AMIA Annu Symp Proc, 2010, 291-5.
- Patterson O, Igo S, Hurdle JF (2010). Automatic acquisition of sublanguage semantic schema: towards the word sense disambiguation of clinical narratives. AMIA Annu Symp Proc, 2010, 612-6.
- Workman TE, Fiszman M, Hurdle JF, Rindflesch TC (2010). Biomedical text summarization to support genetic database curation: using Semantic MEDLINE to create a secondary database of genetic information. J Med Libr Assoc, 98(4), 273-81.
- Meystre SM, Thibault J, Shen S, Hurdle JF, South BR (2010). Textractor: a hybrid system for medications and reason for their prescription extraction from clinical text documents. J Am Med Inform Assoc, 17(5), 559-62.
- Meystre SM, Thibault J, Shen S, Hurdle JF, South BR (2010). Automatically detecting medications and the reason for their prescription in clinical narrative text documents. Stud Health Technol Inform, 160(Pt 2), 944-8.
- Phansalkar S, Hoffman JM, Hurdle JF, Patel VL (2009). Understanding pharmacist decision making for adverse drug event (ADE) detection. J Eval Clin Pract, 15(2), 266-75.
- Lin RS, Horn SD, Hurdle JF, Goldfarb-Rumyantzev AS (12/01/2008). Single and multiple time-point prediction models in kidney transplant outcomes. J Biomed Inform, 41(6), 944-52.
- Hurdle JF, Botkin J, Rindflesch TC (2007). Leveraging semantic knowledge in IRB databases to improve translation science. AMIA Annu Symp Proc, 349-53.
- Penz JF, Wilcox AB, Hurdle JF (2007). Automated identification of adverse events related to central venous catheters. J Biomed Inform, 40(2), 174-182.
- Hurdle JF, Adams S, Brokel J, Chang B, Embi P, Petersen C, Terrazas E, Winkelstein P (2007). A code of professional ethical conduct for the American Medical Informatics Association: an AMIA Board of Directors approved white paper. J Am Med Inform Assoc, 14(4), 391-393.
- Bennett CL, Nebeker JR, Yarnold PR, Tigue CC, Dorr DA, McKoy JM, Edwards BJ, Hurdle JF, West DP, Lau DT, Angelotta C, Weitzman SA, Belknap SM, Djulbegovic B, Tallman MS, Kuzel TM, Benson AB, Evens A, Trifilio SM, Courtney DM, Raisch DW (2007). Evaluation of serious adverse drug reactions: a proactive pharmacovigilance program (RADAR) vs safety activities conducted by the Food and Drug Administration and pharmaceutical manufacturers. Arch Intern Med, 167(10), 1041-9.
- Phansalkar S, Hoffman JM, Nebeker JR, Hurdle JF (2007). Pharmacists versus nonpharmacists in adverse drug event detection: a meta-analysis and systematic review. Am J Health Syst Pharm, 64(8), 842-9.
- Goldfarb-Rumyantzev AS, Hurdle JF, Baird BC, Stoddard G, Wang Z, Scandling JD, Barenbaum LL, Cheung AK (2006). The role of pre-emptive re-transplant in graft and recipient outcome. Nephrol Dial Transplant, 21(5), 1355-64.
- Dorr DA, Phillips WF, Phansalkar S, Sims SA, Hurdle JF (2006). Assessing the difficulty and time cost of de-identification in clinical narratives. Methods Inf Med, 45(3), 246-52.
- Lin SJ, Koford JK, Baird BC, Hurdle JF, Krikov S, Habib AN, Goldfarb-Rumyantzev AS (2005). Effect of donors' intravenous drug use, cigarette smoking, and alcohol dependence on kidney transplant outcome. Transplantation, 80(4), 482-486.
- Goldfarb-Rumyantzev AS, Hurdle JF, Scandling JD, Baird BC, Cheung AK (2005). The role of pretransplantation renal replacement therapy modality in kidney allograft and recipient survival. Am J Kidney Dis, 46(3), 537-49.
- OMalley KJ, Cook KF, Price MD, Wildes KR, Hurdle JF, Ashton CM (2005). Measuring diagnoses: ICD code accuracy. Health Serv Res, 40(5 Pt 2), 1620-39.
- Nebeker JR, Hoffman JM, Weir CR, Bennett CL, Hurdle JF (2005). High rates of adverse drug events in a highly computerized hospital. Arch Intern Med, 165(10), 1111-6.
- Weir CR, Hoffman JM, Nebeker JR, Hurdle JF (2005). Nurses' role in tracking adverse drug events: the impact of provider order entry. Nurs Adm Q, 29(1), 39-44.
- Goldfarb-Rumyantzev A, Hurdle JF, Scandling J, Wang Z, Baird B, Barenbaum L, Cheung AK (2005). Duration of end-stage renal disease and kidney transplant outcome. Nephrol Dial Transplant, 20(1), 167-75.
- Hurdle JF (2004). Can the electronic medical record improve geriatric care? Geriatric Times: an interdisciplinary approach to healthy aging, 5(2), 25-6.
- Nebeker JR, Hurdle JF, Blair BD (2003). Future history: medical informatics in geriatrics. J Gerontol A Biol Sci Med Sci, 58(9), M820-825.
- Weir CR, Hurdle JF, Felgar MA, Hoffman JM, Roth B, Nebeker JR (2003). Direct text entry in electronic progress notes. An evaluation of input errors. Methods Inf Med, 42(1), 61-7.
- Hurdle JF, Weir CR, Roth B, Hoffman J, Nebeker JR (2003). Critical gaps in the world's largest electronic medical record: Ad Hoc nursing narratives and invisible adverse drug events. AMIA Annu Symp Proc, 309-12.
- Nebeker JR, Hurdle JF, Hoffman JM, Roth B, Weir CR, Samore M (2001). Developing a taxonomy for research in adverse drug events: potholes and signposts. Proc AMIA Symp, 493-497.
- Bair B, Toth W, Johnson MA, Rosenberg C, Hurdle JF (1999). Interventions for disruptive behaviors. Use and success. J Gerontol Nurs, 25(1), 13-21.
- Eagon JC, Ortiz E, Zollo KA, Hurdle J, Lincoln MJ (1997). Department of Veterans Affairs, University of Utah consortium participation in the NLM/AHCPR Large Scale Vocabulary Test. Proc AMIA Annu Fall Symp, 565-9.
- Hurdle JF (1997). The synthesis of compact fuzzy-neural circuits. IEEE Transactions on Fuzzy Systems, 5(1), 44-55.
- Hurdle JF (1997). Lightweight fuzzy processes in clinical computing. Artif Intell Med, 11(1), 55-73.
- Eagon JC, Hurdle JF, Lincoln MJ (1996). Inter-rater reliability and review of the VA unresolved narratives. Proc AMIA Annu Fall Symp, 130-4.
- Meystre SM, Savova GK, Kipper-Schuler KC, Hurdle JF (2008). Extracting information from textual documents in the electronic health record: a review of recent research. [Review]. Yearb Med Inform, 128-44.
- Facelli LC, Hurdle JF, Mitchell JA (February 2012). Medical and Bioinformatics. In Ziad O., Abu-Faraj (Eds.), Biomedical Engineering Education & Advanced Bioengineering Learning: Interdisciplinary Concepts. Hershey, PA: IGI GLOBAL.
- Hurdle JF, Brunvand E, Josephson L (1994). Asynchronous VLSI design for neural system implementation. In Delgado-Frias JG, Moore WR (Eds.), VLSI and Neural Networks and Artificial Intelligence. New York: Plenum Press.
- Hurdle JF, Josephson L, Brunvand E (1994). Reliable interfacing of self-timed FPGA-based hybrid neural/rule-based classifiers to synchronous co-processors. In Luk W, Moore WR (Eds.), More FPGAs. Oxford: Abingdon Books.
- Doing-Harris K, Igo S, Hurdle JF (August 2014). An Innovation in the Integration of Cognitive Science and NLP: A Multi-level, Multi-path Approach. Quebec City, CA: 36th Annual Conference of the Cognitive Science Society.
- Chidambaram VC, Brewster PJ, Tran T, Jordan KC, Hurdle JF (2014). qDIET: Toward Calculating HEI Scores From Grocery Store Sales Data. Proc NJNDC 2014, Portland, OR: NNDC.
- Doing-Harris K, Patterson O, Igo S, Hurdle J (2013). Document Sublanguage Clustering to Detect Medical Specialty in Cross-institutional Clinical Texts. Proc ACM Int Workshop Data Text Min Biomed Inform, United States, 2013, 9-12.
- Doing-Harris K, Patterson O, Igo S, Hurdle JF (2013). Document Sublanguage Clustering to Detect Medical Specialty in Cross-institutional Clinical Texts. ACM International Conference on Information and Knowledge Management (CIKM 2013), San Francisco: DTMBIOI '13 Proceedings of the 7th International Workshop on Data and Text Mining in Biomedical Informatics, 9-12.
- Hurdle JF, Satsang S, Hoffman JM (2004). VistA Progress Notes as a Source for Adverse Drug Event Signals. VA HSR&D Ann Meet, VA Health Services and Research Annual Meeting.
- Hurdle JF (1993). Self-timed neural model implementation: an example using CMAC. Proc. of the Hawaii International Conference on Systems Sciences, I, 369-378.
- Josephson L, Brunvand E, Gopalkrishnan G, Hurdle JF (1993). Reliable interface design for combining asynchronous and synchronous circuits. Proc Fifth Annual NASA Symposium on VLSI Design, 10, 1.
- Hurdle JF, Conwell PR, Brunvand E (1993). Robust neural classifier circuits using asynchronous design. Proc Fifth Annual NASA Symp on VLSI Design, 8, 1.
- Hurdle JF, Josephson L, Brunvand E, Gopalakrishnan G (1992). Asynchronous models for large scale neurocomputing applications. Proc. of the Fifth International Conference on Neural Networks and Their Applications, 577-588.
- Tran LT, Brewster PJ, Chidambaram V, Hurdle JF (2014). An Estimation Model of the Healthy Eating Index 2010 to Measure the Dietary Quality of Grocery Purchases. (poster) [Abstract].
- Mineau GP, Courdy SJ, Haroldsen C, Hammer A, Spigle C, Schaefer, C, Pimentel R, Fraser A, Barlow J, Hurdle JF (2010). Finding subjects for clinical research using state-wide, multi-source data: the utah population database limited cohort tool [Abstract].
- Patterson O, Hurdle JF (2009). Building Domain Information Schemas using semisupervised machine learning [Abstract].
- Rocha RA, Hurdle JF, Matney S, Narus SP, Meystre S, LaSalle B, Deshmukh V, Hunter C, Mineau GP, Facelli JC, Mitchell JA (2008). Utah's statewide informatics platform for translational and clinical science [Abstract]. AMIA Annu Symp Proc, 1114.
- Phansalkar S, South BR, Hoffman JM, Hurdle JF (2007). Looking for a needle in the haystack? A case for detecting adverse drug events (ADE) in clinical notes [Abstract]. AMIA Annu Symp Proc, 1077.
- Tang H, Goldfarb-Rumyantzev A, Hunter C, Poynton M, Tu M, Baird B, Krikov S, Koford J, Hurdle JF (2007). Validating prediction models for kidney transplant outcome using local data [Abstract]. AMIA Annu Symp Proc, 1128.
- Phansalkar S, Patel VL, Hoffman JM, Hurdle JF (2006). Use of verbal protocol analysis for identification of ADE signals [Abstract]. AMIA Annu Symp Proc.
- Goldfarb-Rumyantzev A, Kirkov S, Cheung A, Hurdle JF (2004). Pre-transplant dialysis modality determines kidney transplant outcome [Abstract]. American Society of Nephrology.
- Brewster PJ, Hurdle JF (June, 2011). Dietary Pattern Analysis Using Electronic Grocery Transaction Data. Poster session presented at National Library of Medicine Annual Training Conference 2011, Bethesda, MD.
- Brewster P, Brinkerhoff KM, Hurdle J (04/08/2011). Beyond Self-Reported Measures: Using Electronic Grocery Data in Dietary Assessment. Poster session presented at 35th National Nutrient Databank Conference, Washington, DC.
- Zafar N, Bray B, Hurdle JF, Staggers N (06/15/2010). A Call For Re-design: Usability Evaluation Of a Vendor’s ED CPOE. Poster session presented at NLM Informatics Training Conference 2010, Dnevr, CO.
- Angelotta C, Dorr D, West D, Burdon R, Lagman J, Workman D, Hurdle JF, Belknap SM, Bennett CL (2007). New IRB guidelines: Will cancer patient safety improve?. Poster session presented at Amer Soc Clinical Onco, Chicago, IL.
- Hurdle JF, Satsangi S, Hoffman JM (2005). Detecting ADEs in Diverse Domain and Author Clinical Narratives. Poster session presented at NLM Trainee Annual Meeting.
- Hurdle JF, Weir CR, Nebeker JR (2003). Inter-coder reliability of ICD9 assignment: flaws in an important diversity assessment tool. Poster session presented at VA Health Services and Research Annual Meeting.
- Hurdle JF, Roth BS, Nebeker JR, Hoffman JM, Weir CR (2002). Workshop: Improving patient safety: advances in ADE research and Surveillance. Poster session presented at VA Health Services and Research Annual Meeting.
- Weir CR, Hurdle JF, Nebeker JR, Hoffman JM (2002). Age as a risk factor for injury and error associated with drug use. Poster session presented at VA Health Services and Research Annual Meeting.
- Hurdle JF, Weir CR, Nebeker JR (2001). Inpatient adverse drug event monitoring: efficacy and cost of automated and non-automated methods.