Testing in the Wild: The Social and Organisational Dimensions of Real World Practice
Loading...
Fulltext URI
Document type
Text/Journal Article
Additional Information
Journal Title
Journal ISSN
Volume Title
Publisher
Springer
Abstract
Testing is a key part of any systems engineering project. There is an extensive literature on testing, but very little that focuses on how testing is carried out in real-world circumstances. This is partly because current practices are often seen as unsophisticated and ineffective. We believe that by investigating and characterising the real-world work of testing we can help question why such ‘bad practices’ occur and how improvements might be made. We also argue that the testing literature is too focused on technological issues when many of the problems, and indeed strengths, have as much do with work and organisation. In this paper we use empirical examples from four systems engineering projects to demonstrate how and in what ways testing is a cooperative activity. In particular we demonstrate the ways in which testing is situated within organisational work and satisfices organisational and marketplace demands.
Description
Keywords
dependability, ethnography, ethnomethodology, organisational issues, software development, systems testing, work practices
Citation
URI
Collections
Endorsement
Review
Supplemented By
Referenced By
Number of citations to item: 62
- Anca Deak, Tor Stalhane (2013): Organization of Testing Activities in Norwegian Software Companies, In: 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops, doi:10.1109/icstw.2013.18
- Nils Wild, Horst Lichter, Peter Kehren (2020): Test Automation Challenges for Application Landscape Frameworks, In: 2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), doi:10.1109/icstw50294.2020.00059
- Liane Colonna, Stockholm Law Submitter (2023): The AI Act’s Research Exemption: A Mechanism for Regulatory Arbitrage?, In: SSRN Electronic Journal, doi:10.2139/ssrn.4575971
- Jalal Mahmud, Allen Cypher, Eben Haber, Tessa Lau (2012): Design and industrial evaluation of a tool supporting semi-automated website testing, In: Software Testing, Verification and Reliability 1(24), doi:10.1002/stvr.1484
- Dave Cliff, Linda Northrop (2012): The Global Financial Markets: An Ultra-Large-Scale Systems Perspective, In: Lecture Notes in Computer Science, doi:10.1007/978-3-642-34059-8_2
- Mika V. Mäntylä, Juha Itkonen (2014): How are software defects found? The role of implicit defect detection, individual responsibility, documents, and knowledge, In: Information and Software Technology 12(56), doi:10.1016/j.infsof.2013.12.005
- Hina Shah, Mary Jean Harrold (2010): Studying human and social aspects of testing in a service-based software company, In: Proceedings of the 2010 ICSE Workshop on Cooperative and Human Aspects of Software Engineering, doi:10.1145/1833310.1833327
- Degif Teka, Yvonne Dittrich, Mesfin Kifle, Carmelo Ardito, Rosa Lanzilotti (2017): User Involvement and Usability Evaluation in Ethiopian Software Organizations, In: THE ELECTRONIC JOURNAL OF INFORMATION SYSTEMS IN DEVELOPING COUNTRIES 1(83), doi:10.1002/j.1681-4835.2017.tb00616.x
- Götz Hoeppe (2021): Encoding Collective Knowledge, Instructing Data Reusers: The Collaborative Fixation of a Digital Scientific Data Set, In: Computer Supported Cooperative Work (CSCW) 4(30), doi:10.1007/s10606-021-09407-2
- Tabata Pérez Rentería y Hernández, Nicola Marsden (2021): Offshore Software Testing in the Automotive Industry, In: Research Anthology on Recent Trends, Tools, and Implications of Computer Programming, doi:10.4018/978-1-7998-3016-0.ch070
- Ronnie Edson de Souza Santos, Cleyton Vanut Cordeiro de Magalhaes, Jorge da Silva Correia-Neto, Fabio Queda Bueno da Silva, Luiz Fernando Capretz, Rodrigo Souza (2017): Would You Like to Motivate Software Testers? Ask Them How, In: 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), doi:10.1109/esem.2017.16
- Hina Shah, Mary Jean Harrold, Nancy J. Nersessian (2011): Studying the Influence of Culture on Outsourced, Offshored Software-Testing Practice: An Ethnographic Approach, In: 2011 IEEE Sixth International Conference on Global Software Engineering Workshop, doi:10.1109/icgse-w.2011.27
- John Fowler, Mark Zachry (2024): VizCare: Using UCD Techniques to Impact the Child Welfare System, In: 2024 IEEE International Professional Communication Conference (ProComm), doi:10.1109/procomm61427.2024.00029
- Raphael Pham, Stephan Kiesling, Olga Liskin, Leif Singer, Kurt Schneider (2014): Enablers, inhibitors, and perceptions of testing in novice software teams, In: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, doi:10.1145/2635868.2635925
- Samir Passi, Phoebe Sengers (2020): Making data science systems work, In: Big Data & Society 2(7), doi:10.1177/2053951720939605
- Emelie Engström, Per Runeson (2010): A Qualitative Survey of Regression Testing Practices, In: Lecture Notes in Computer Science, doi:10.1007/978-3-642-13792-1_3
- Pradeep Waychal, Luiz Fernando Capretz, Jingdong Jia, Daniel Varona, Yadira Lizama (2021): Practitioners’ Testimonials about Software Testing, In: 2021 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), doi:10.1109/saner50967.2021.00070
- Claire Hill, Cathy Creswell, Sarah Vigerland, Maaike H. Nauta, Sonja March, Caroline Donovan, Lidewij Wolters, Susan H. Spence, Jennifer L. Martin, Lori Wozney, Lauren McLellan, Leonie Kreuze, Karen Gould, Maral Jolstedt, Martina Nord, Jennifer L. Hudson, Elisabeth Utens, Jeroen Ruwaard, Casper Albers, Muniya Khanna, Anne Marie Albano, Eva Serlachius, Stefan Hrastinski, Philip C. Kendall (2018): Navigating the development and dissemination of internet cognitive behavioral therapy (iCBT) for anxiety disorders in children and young people: A consensus statement with recommendations from the #iCBTLorentz Workshop Group, In: Internet Interventions, doi:10.1016/j.invent.2018.02.002
- Nilam Kaushik, Mazeiar Salehie, Ladan Tahvildari, Sen Li, Mark Moore (2011): Dynamic Prioritization in Regression Testing, In: 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops, doi:10.1109/icstw.2011.68
- Louise Petersen Matjeka, Dag Svanæs, Alf Inge Wang (2022): Turning Eight Family Homes Into Interactive, Pervasive Playgrounds During the COVID-19 Pandemic Lockdown, In: Frontiers in Computer Science, doi:10.3389/fcomp.2022.822337
- Per Runeson, Emelie Engström (2012): Regression Testing in Software Product Line Engineering, In: Advances in Computers, doi:10.1016/b978-0-12-396535-6.00007-7
- Josh Tenenberg, Wolff-Michael Roth, David Socha (2015): From I-Awareness to We-Awareness in CSCW, In: Computer Supported Cooperative Work (CSCW) 4-5(25), doi:10.1007/s10606-014-9215-0
- Tanjila Kanij, Robert Merkel, John Grundy (2015): An Empirical Investigation of Personality Traits of Software Testers, In: 2015 IEEE/ACM 8th International Workshop on Cooperative and Human Aspects of Software Engineering, doi:10.1109/chase.2015.7
- Dusica Marijan (2012): A Review of Two Experiences from Applying Model Based Testing in Practice, In: 2012 IEEE 23rd International Symposium on Software Reliability Engineering Workshops, doi:10.1109/issrew.2012.78
- Liane Colonna (2023): The AI Act’s Research Exemption: A Mechanism for Regulatory Arbitrage?, In: YSEC Yearbook of Socio-Economic Constitutions, doi:10.1007/16495_2023_59
- Tanjila Kanij, Robert Merkel, John Grundy (2013): An empirical study of the effects of personality on software testing, In: 2013 26th International Conference on Software Engineering Education and Training (CSEE&T), doi:10.1109/cseet.2013.6595255
- Anca Deak (2012): Understanding Socio-technical Factors Influencing Testers in Software Development Organizations, In: 2012 IEEE 36th Annual Computer Software and Applications Conference, doi:10.1109/compsac.2012.103
- Juha Itkonen, Mika V. Mantyla, Casper Lassenius (2013): The Role of the Tester's Knowledge in Exploratory Software Testing, In: IEEE Transactions on Software Engineering 5(39), doi:10.1109/tse.2012.55
- Kristian Wiklund, Sigrid Eldh, Daniel Sundmark, Kristina Lundqvist (2017): Impediments for software test automation: A systematic literature review, In: Software Testing, Verification and Reliability 8(27), doi:10.1002/stvr.1639
- Stefan Mohacsi, Rudolf Ramler (2018): Why Software Testing Fails: Common Pitfalls Observed in a Critical Smart Metering Project, In: Lecture Notes in Business Information Processing, doi:10.1007/978-3-030-05767-1_6
- Vitalis Ndume, Yaw Nkansah-Gyekye, Jesuk Ko (2014): A web service approach for integrating e-Health Data in a low bandwidth environment, In: Proceedings of the 2nd Pan African International Conference on Science, Computing and Telecommunications (PACT 2014), doi:10.1109/scat.2014.7055133
- Kristian Wiklund, Monika Wiklund (2018): The Next Level of Test Automation: What About the Users?, In: 2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), doi:10.1109/icstw.2018.00045
- Julia Prior (2013): A sense of working there, In: Proceedings of the 25th Australian Computer-Human Interaction Conference: Augmentation, Application, Innovation, Collaboration, doi:10.1145/2541016.2541089
- Anca Deak, Tor Stålhane, Guttorm Sindre (2016): Challenges and strategies for motivating software testing personnel, In: Information and Software Technology, doi:10.1016/j.infsof.2016.01.002
- Tabata Pérez Rentería y Hernández, Nicola Marsden (2017): Offshore Software Testing in the Automotive Industry, In: International Journal of Information Technology Project Management 4(8), doi:10.4018/ijitpm.2017100101
- Matt Osborne, Richard Hawkins, Mark Nicholson, Rob Alexander (2024): Understanding safety engineering practice: Comparing safety engineering practice as desired, as required, and as observed, In: Safety Science, doi:10.1016/j.ssci.2024.106424
- Tone Bratteteig, Ina Wagner (2016): Unpacking the Notion of Participation in Participatory Design, In: Computer Supported Cooperative Work (CSCW) 6(25), doi:10.1007/s10606-016-9259-4
- Hina Shah, Mary Jean Harrold, Saurabh Sinha (2014): Global software testing under deadline pressure: Vendor-side experiences, In: Information and Software Technology 1(56), doi:10.1016/j.infsof.2013.04.005
- Moritz Beller, Georgios Gousios, Annibale Panichella, Sebastian Proksch, Sven Amann, Andy Zaidman (2019): Developer Testing in the IDE: Patterns, Beliefs, and Behavior, In: IEEE Transactions on Software Engineering 3(45), doi:10.1109/tse.2017.2776152
- Simone Romano, Davide Fucci, Giuseppe Scanniello, Burak Turhan, Natalia Juristo (2017): Findings from a multi-method study on test-driven development, In: Information and Software Technology, doi:10.1016/j.infsof.2017.03.010
- Claire Hill, Tessa Reardon, Lucy Taylor, Cathy Creswell (2022): Online Support and Intervention for Child Anxiety (OSI): Development and Usability Testing, In: JMIR Formative Research 4(6), doi:10.2196/29846
- Hina Shah, Mary Jean Harrold (2013): Culture and Testing: What is the Relationship?, In: 2013 IEEE 8th International Conference on Global Software Engineering, doi:10.1109/icgse.2013.15
- Helen Sharp, Yvonne Dittrich, Cleidson R. B. de Souza (2016): The Role of Ethnographic Studies in Empirical Software Engineering, In: IEEE Transactions on Software Engineering 8(42), doi:10.1109/tse.2016.2519887
- Ilkka Tervonen, Antti Haapalahti, Lasse Harjumaa, Jouni Simila (2013): Outsourcing Software Testing: A Case Study in the Oulu Area, In: 2013 13th International Conference on Quality Software, doi:10.1109/qsic.2013.53
- Nasir Mehmood Minhas, Kai Petersen, Jürgen Börstler, Krzysztof Wnuk (2020): Regression testing for large-scale embedded software development – Exploring the state of practice, In: Information and Software Technology, doi:10.1016/j.infsof.2019.106254
- Muhammad Shah Jahan, Muhammad Talha Riaz, Kashif, Muhammad Abbas (2019): Software Testing Practices in IT Industry of Pakistan, In: Proceedings of the 6th Conference on the Engineering of Computer Based Systems, doi:10.1145/3352700.3352724
- Anca Deak (2013): Understanding the Influence of Social and Technical Factors Testers in Software Organizations, In: 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation, doi:10.1109/icst.2013.76
- Mika V. Mäntylä, Juha Itkonen, Joonas Iivonen (2011): Who tested my software? Testing as an organizationally cross-cutting activity, In: Software Quality Journal 1(20), doi:10.1007/s11219-011-9157-4
- Fredrik Asplund (2019): Exploratory testing: Do contextual factors influence software fault identification?, In: Information and Software Technology, doi:10.1016/j.infsof.2018.11.003
- Eric Monteiro, Neil Pollock, Ole Hanseth, Robin Williams (2012): From Artefacts to Infrastructures, In: Computer Supported Cooperative Work (CSCW) 4-6(22), doi:10.1007/s10606-012-9167-1
- Luiz Fernando Capretz, Saad Harous, Ali Bou Nassif (2022): What UAE Software Students Think About Software Testing: A Replicated Study, In: Advances in Intelligent Systems and Computing, doi:10.1007/978-3-031-16684-6_7
- Hina Shah, Saurabh Sinha, Mary Jean Harrold (2011): Outsourced, Offshored Software-Testing Practice: Vendor-Side Experiences, In: 2011 IEEE Sixth International Conference on Global Software Engineering, doi:10.1109/icgse.2011.32
- Ashley D Radomski, Lori Wozney, Patrick McGrath, Anna Huguet, Lisa Hartling, Michele P Dyson, Kathryn Bennett, Amanda S Newton (2019): Design and Delivery Features That May Improve the Use of Internet-Based Cognitive Behavioral Therapy for Children and Adolescents With Anxiety: A Realist Literature Synthesis With a Persuasive Systems Design Perspective, In: Journal of Medical Internet Research 2(21), doi:10.2196/11128
- Anca Deak (2014): What Characterizes a Good Software Tester? – A Survey in Four Norwegian Companies, In: Lecture Notes in Computer Science, doi:10.1007/978-3-662-44857-1_11
- Xin Xia, David Lo, Pavneet Singh Kochhar, Zhenchang Xing, Xinyu Wang, Shanping Li (2015): Experience report: An industrial experience report on test outsourcing practices, In: 2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE), doi:10.1109/issre.2015.7381830
- Bruce R. Maxim, Marouane Kessentini (2016): An introduction to modern software quality assurance, In: Software Quality Assurance, doi:10.1016/b978-0-12-802301-3.00002-8
- Moritz Beller, Georgios Gousios, Annibale Panichella, Andy Zaidman (2015): When, how, and why developers (do not) test in their IDEs, In: Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, doi:10.1145/2786805.2786843
- Edward Meade, Emma O’Keeffe, Niall Lyons, Dean Lynch, Murat Yilmaz, Ulas Gulec, Rory V. O’Connor, Paul M. Clarke (2019): The Changing Role of the Software Engineer, In: Communications in Computer and Information Science, doi:10.1007/978-3-030-28005-5_53
- Claire Hill, Tessa Reardon, Lucy Taylor, Cathy Creswell (2021): Online Support and Intervention for Child Anxiety (OSI): Development and Usability Testing (Preprint), doi:10.2196/preprints.29846
- Fabio Massacci (2010): Infringo ergo sum, In: Proceedings of the FSE/SDP workshop on Future of software engineering research, doi:10.1145/1882362.1882411
- Yadira Lizama, Daniel Varona, Pradeep Waychal, Luiz Fernando Capretz (2019): The Unpopularity of the Software Tester Role Among Software Practitioners: A Case Study, In: Springer Series in Reliability Engineering, doi:10.1007/978-3-030-36518-9_7
- Mujtaba Hassan, Muzammil Hussain, Salah-u-din Ayubi, Maham Irfan (2019): A Policy Recommendations Framework To Resolve Global Software Development Issues, In: 2019 International Conference on Innovative Computing (ICIC), doi:10.1109/icic48496.2019.8966719