|
UPSI Digital Repository (UDRep)
|
|
| ||||||||||||||||||||||||||
| Abstract : Universiti Pendidikan Sultan Idris |
| Context: A plethora of models are available for open-source software (OSS) usability evaluation. However, these models lack consensus between scholars as well as standard bodies on a specific set of usability evaluation criteria. Retaining irrelevant criteria and omitting essential ones will mislead the direction of the usability evaluation. Objective: This study introduces a three-step method to develop a usability evaluation model in the context of OSS. Method: The fuzzy Delphi method has been employed to unify the usability evaluation criteria in the context of OSS. The first step in the method is the usability criteria analysis, which involves redefining and restructuring all collected usability criteria reported in the literature. The second step is fuzzy Delphi analysis, which includes the design and validates the fuzzy Delphi instrument and the utilisation of the fuzzy Delphi method to analyse the fuzziness consensus of experts' opinions on the usability evaluation criteria. The third step is the proposal of the OSS usability evaluation model. Results: A total of 124 usability criteria were identified, redefined, and restructured by creating groups of related meaning criteria. The result of the groupings generated 11 main criteria; the findings of the fuzzy Delphi narrowed down the criteria to only seven. The final set of criteria was sent back to the panellists for reconsideration of their responses. The panellists verified that these criteria are suitable in the evaluation of the usability of OSS. Discussion: The empirical analysis confirmed that the proposed evaluation model is acceptable in assessing the usability of OSS. Therefore, this model can be used as a reference metric for OSS usability evaluation which will have a practical benefit for the community in public and private organisations in helping the decision-maker to select the best OSS software package amongst the alternatives. ? 2020 |
| References |
(1992). , 1061-1992. Retrieved from www.scopus.com (2001). Retrieved from www.scopus.com (2001). Retrieved from www.scopus.com IEEE std 1061-1998 - IEEE standard for a software quality metrics methodology. (1998). IEEE Std.1061-1998: IEEE Standard for A Software Quality Metrics Methodology, Retrieved from www.scopus.com ISO 9241-11. (2018). ergonomics of human–system interaction – part 11: Usability: Definitions and concepts. (2018). Retrieved from www.scopus.com Abran, A., Khelifi, A., Suryn, W., & Seffah, A. (2003). Usability meanings and interpretations in ISO standards. Software Quality Journal, 11(4), 325-338. doi:10.1023/A:1025869312943 Adler, M., & Ziglio, E. (1996). Gazing into the oracle: The delphi method and its application to social policy and public health. Gazing into the Oracle: The Delphi Method and its Application to Social Policy and Public Health, Retrieved from www.scopus.com Ahmad, Z., Muhidin, M., Wasli, P., Salihin, M., & Mohd, H. (2014). Fuzzy delphi analysis for future environmental education using interactive animation. 2nd International Seminar Teaching Excellence and Innovation 25 February 2014, Retrieved from www.scopus.com Alonso-Ríos, D., Vázquez-García, A., Mosqueira-Rey, E., & Moret-Bonillo, V. (2010). Usability: A critical analysis and a taxonomy. International Journal of Human-Computer Interaction, 26(1), 53-74. doi:10.1080/10447310903025552 Al-Rawashdeh, T. A. (2015). Evaluating open source software usability using a multistage fuzzy model approach. International Review on Computers and Software, 10(10), 1018-1026. doi:10.15866/irecos.v10i10.7668 Anthony Jnr, B. (2019). Validating the usability attributes of AHP-software risk prioritization model using partial least square-structural equation modeling. Journal of Science and Technology Policy Management, 10(2), 404-430. doi:10.1108/JSTPM-06-2018-0060 Bahamdain, S. S. (2015). Open source software (OSS) quality assurance: A survey paper. Paper presented at the Procedia Computer Science, , 56(1) 459-464. doi:10.1016/j.procs.2015.07.236 Retrieved from www.scopus.com Baytiyeh, H. (2011). Work in progress - open source usability evaluation: The case of moodle. Paper presented at the Proceedings - Frontiers in Education Conference, FIE, doi:10.1109/FIE.2011.6142798 Retrieved from www.scopus.com Bevan, N. (2001). International standards for HCl and usability. International Journal of Human Computer Studies, 55(4), 533-552. doi:10.1006/ijhc.2001.0483 Bodjanova, S. (2006). Median alpha-levels of a fuzzy number. Fuzzy Sets and Systems, 157(7), 879-891. doi:10.1016/j.fss.2005.10.015 Bolarinwa, O. A. (2015). Principles and methods of validity and reliability testing of questionnaires used in social and health science researches. Niger Postgrad Med J, 22(4), 195-201. Retrieved from www.scopus.com Booth, P. (2014). Retrieved from www.scopus.com Borsci, S., Federici, S., Malizia, A., & De Filippis, M. L. (2019). Shaking the usability tree: Why usability is not a dead end, and a constructive way forward. Behaviour and Information Technology, 38(5), 519-532. doi:10.1080/0144929X.2018.1541255 Bryman, A., & Cramer, D. (1997). Quantitative data analysis with SPSS for windows. Quantitative Data Analysis with SPSS for Windows, Retrieved from www.scopus.com Capiluppi, A., Boldyreff, C., Beecher, K., & Adams, P. J. (2009). Quality factors and coding standards - a comparison between open source forges. Electronic Notes in Theoretical Computer Science, 233(C), 89-103. doi:10.1016/j.entcs.2009.02.063 Ćetin, G., & Mehmet, G. (2008). A measurement based framework for assessment of usability-centricness of open source software projects. Paper presented at the SITIS 2008 - Proceedings of the 4th International Conference on Signal Image Technology and Internet Based Systems, 585-592. doi:10.1109/SITIS.2008.106 Retrieved from www.scopus.com Chang, P. -., Hsu, C. -., & Chang, P. -. (2011). Fuzzy delphi method for evaluating hydrogen production technologies. International Journal of Hydrogen Energy, 36(21), 14172-14179. doi:10.1016/j.ijhydene.2011.05.045 Chen, F. -., Chen, J. -., Wang, J. -., & Tai, D. W. -. (2017). Using fuzzy delphi method to construct digital literacy competences for junior high school students. International Journal of Information and Education Technology, 7(9), 686-689. Retrieved from www.scopus.com Cheng, C. -., & Lin, Y. (2002). Evaluating the best main battle tank using fuzzy decision theory with linguistic criteria evaluation. European Journal of Operational Research, 142(1), 174-186. doi:10.1016/S0377-2217(01)00280-6 Chu, H. -., & Hwang, G. -. (2008). A delphi-based approach to developing expert systems with the cooperation of multiple experts. Expert Systems with Applications, 34(4), 2826-2840. doi:10.1016/j.eswa.2007.05.034 Chuang, C. -. L. A. L. Z. -. (2012). Using fuzzy delphi method and fuzzy AHP for evaluation structure of the appeal of taiwan's coastal wetlands ecotourism. Business, Economics, and Financial Sci., Manag., 2012, 347-358. Retrieved from www.scopus.com Côté, M. -., Suryn, W., & Georgiadou, E. (2007). In search for a widely applicable and accepted software quality model for software quality engineering. Software Quality Journal, 15(4), 401-416. doi:10.1007/s11219-007-9029-0 Crowston, K., Annabi, H., Howison, J., & Masango, C. (2004). Effective work practices for software engineering: Free/libre open source software development. Paper presented at the WISER 2004 - ACM Workshop on Interdisciplinary Software Engineering Research, 18-26. Retrieved from www.scopus.com Crowston, K., & Howison, J. (2005). The social structure of free and open source software development. First Monday, 10(2) doi:10.5210/fm.v10i2.1207 Dajani, J. S., Sincoff, M. Z., & Talley, W. K. (1979). Stability and agreement criteria for the termination of delphi studies. Technological Forecasting and Social Change, 13(1), 83-90. doi:10.1016/0040-1625(79)90007-6 Dapari, R., Ismail, H., Ismail, R., & Ismail, N. H. (2017). Application of fuzzy delphi in the selection of COPD risk factors among steel industry workers. Tanaffos, 16(1), 46-52. Retrieved from www.scopus.com Dawood, K. A., Sharif, K. Y., Zaidan, A. A., Abd Ghani, A. A., Zulzalil, H. B., & Zaidan, B. B. (2019). Mapping and analysis of open source software (OSS) usability for sustainable OSS product. IEEE Access, 7, 65913-65933. doi:10.1109/ACCESS.2019.2914368 Devon, H. A., Block, M. E., Moyle-Wright, P., Ernst, D. M., Hayden, S. J., Lazzara, D. J., . . . Kostas-Polston, E. (2007). A psychometric toolbox for testing validity and reliability. Journal of Nursing Scholarship, 39(2), 155-164. doi:10.1111/j.1547-5069.2007.00161.x Dix, A., Finlay, J., Abowd, G., & Beale, R. (1993). Human-Computer Interaction, Retrieved from www.scopus.com Donyaee, M. (2001). Retrieved from www.scopus.com Dube, M. R., & Dixit, S. K. (2012). Modeling theories and model transformation scenario for complex system development. Int.J.Comput.Appli., 38(15), 11-18. Retrieved from www.scopus.com Falzarano, M., & Pinto Zipp, G. (2013). Seeking consensus through the use of the delphi technique in health sciences research. Journal of Allied Health, 42(2), 99-105. Retrieved from www.scopus.com Feng, A. L., Wesely, N. C., Hoehle, L. P., Phillips, K. M., Yamasaki, A., Campbell, A. P., . . . Sedaghat, A. R. (2017). A validated model for the 22-item sino-nasal outcome test subdomain structure in chronic rhinosinusitis. International Forum of Allergy and Rhinology, 7(12), 1140-1148. doi:10.1002/alr.22025 Fenton, N. E., & Neil, M. (2000). Software metrics: Roadmap. Paper presented at the Proceedings of the Conference on the Future of Software Engineering, ICSE 2000, 357-370. doi:10.1145/336512.336588 Retrieved from www.scopus.com Fernandez, A., Insfran, E., & Abrahão, S. (2011). Usability evaluation methods for the web: A systematic mapping study. Information and Software Technology, 53(8), 789-817. doi:10.1016/j.infsof.2011.02.007 Fernández-Pérez, Y., Febles-Estrada, A., Cruz, C., & Verdegay, J. L. (2018). Fuzzy multi-criteria decision making methods applied to usability software assessment: An annotated bibliography doi:10.1007/978-3-319-69989-9_11 Retrieved from www.scopus.com Fleck, M. P., Chachamovich, E., & Trentini, C. (2006). Development and validation of the portuguese version of the WHOQOL-OLD module. Revista De Saude Publica, 40(5), 785-791. doi:10.1590/s0034-89102006000600007 FrØkjaer, E., Hertzum, M., & Hornbæk, K. (2000). Measuring usability: Are effectiveness, efficiency, and satisfaction really correlated? Paper presented at the Conference on Human Factors in Computing Systems - Proceedings, 345-352. doi:10.1145/332040.332455 Retrieved from www.scopus.com Gupta, D., & Ahlawat, A. K. (2017). Usability feature selection via MBBAT: A novel approach. Journal of Computational Science, 23, 195-203. doi:10.1016/j.jocs.2017.06.005 Gupta, D., Ahlawat, A. K., Sharma, A., & Rodrigues, J. (2020). Retrieved from www.scopus.com Habibi, A., Jahantigh, F. F., & Sarafrazi, A. (2015). Fuzzy delphi technique for forecasting and screening items. Asian Journal of Research in Business Economics and Management, 5(2), 130-143. Retrieved from www.scopus.com Harlen, W. (1999). Purposes and procedures for assessing science process skills. Assessment in Education, 6(1), 129-144. Retrieved from www.scopus.com Harrison, R., Flood, D., & Duce, D. (2013). Usability of mobile applications: Literature review and rationale for a new usability model. Journal of Interaction Science, 1(1), 1-16. Retrieved from www.scopus.com Hauge, Ø. (2007). Master's thesis. Retrieved from www.scopus.com Hauge, Ø., Ayala, C., & Conradi, R. (2010). Adoption of open source software in software-intensive organizations - A systematic literature review. Information and Software Technology, 52(11), 1133-1154. doi:10.1016/j.infsof.2010.05.008 Hedberg, H., Iivari, N., Rajanen, M., & Harjumaa, L. (2007). Assuring quality and usability in open source software development. Paper presented at the First International Workshop on Emerging Trends in FLOSS Research and Development, FLOSS'07, 2-6. doi:10.1109/FLOSS.2007.2 Retrieved from www.scopus.com Hedberg, H., Iivari, N., Rajanen, M., & Harjumaa, L. (2007). Assuring quality and usability in open source software development. Paper presented at the First International Workshop on Emerging Trends in FLOSS Research and Development, FLOSS'07, 2-6. doi:10.1109/FLOSS.2007.2 Retrieved from www.scopus.com Heiko, A. (2012). Consensus measurement in delphi studies: Review. And Implications for Future Quality Assurance.Technological Forecasting and Social Change, 79(8), 1525-1536. Retrieved from www.scopus.com Hill, K. Q., & Fowles, J. (1975). The methodological worth of the delphi forecasting technique. Technological Forecasting and Social Change, 7(2), 179-192. doi:10.1016/0040-1625(75)90057-8 Hsu, Y. -., Lee, C. -., & Kreng, V. B. (2010). The application of fuzzy delphi method and fuzzy AHP in lubricant regenerative technology selection. Expert Systems with Applications, 37(1), 419-425. doi:10.1016/j.eswa.2009.05.068 Iivari, N. (2008). Usability in open source software development - an interpretive case study. Paper presented at the 16th European Conference on Information Systems, ECIS 2008, Retrieved from www.scopus.com ISO, I. (1998). Retrieved from www.scopus.com ISO, I. (1991). ISO 9126/ISO, IEC (hrsg.): International standard ISO/IEC 9126: Information technology-software product evaluation. Qual.Charact.Guidel.use, , 12-15. Retrieved from www.scopus.com ISO, I., & Std, I. (2001). Retrieved from www.scopus.com Joia, L. A., & dos Santos Vinhais, J. C. (2017). From closed source to open source software: Analysis of the migration process to open office. Journal of High Technology Management Research, 28(2), 261-272. doi:10.1016/j.hitech.2017.10.008 Jones, H., & Twiss, B. C. (1978). Retrieved from www.scopus.com Kamei, Y., Matsumoto, T., Yamashita, K., Ubayashi, N., Iwasaki, T., & Takayama, S. (2018). Studying the cost and effectiveness of oss quality assessment models: An experience report of fujitsu QNET. IEICE Transactions on Information and Systems, E101D(11), 2744-2753. doi:10.1587/transinf.2018EDP7163 Kengeri, R., Seals, C. D., Harley, H. D., Reddy, H. P., & Fox, E. A. (1998). Usability study of digital libraries: ACM, IEEE-CS, NCSTRL, NDLTD. International Journal on Digital Libraries, 2(2-3), 157-169. Retrieved from www.scopus.com Khajouei, R., Ameri, A., & Jahani, Y. (2018). Evaluating the agreement of users with usability problems identified by heuristic evaluation. International Journal of Medical Informatics, 117, 13-18. doi:10.1016/j.ijmedinf.2018.05.012 Khosravi, K., & Guéhéneuc, Y. -. (2005). Open issues with quality models. Proceedings of the 9th QAOOSE Workshop(ECOOP), Retrieved from www.scopus.com Kiah, M. L. M., Haiqi, A., Zaidan, B. B., & Zaidan, A. A. (2014). Open source EMR software: Profiling, insights and hands-on analysis. Computer Methods and Programs in Biomedicine, 117(2), 360-382. doi:10.1016/j.cmpb.2014.07.002 Kim, S. W., & Lee, M. G. (2008). Validation of an evaluation model for learning management systems: Original article. Journal of Computer Assisted Learning, 24(4), 284-294. doi:10.1111/j.1365-2729.2007.00260.x Lacerda, T. C., & Von Wangenheim, C. G. (2017). Systematic literature review of usability capability/maturity models. Computer Standards & Interfaces, 55, 95-105. Retrieved from www.scopus.com Lam, K. W., Hassan, A., Sulaiman, T., & Kamarudin, N. (2018). Evaluating the face and content validity of an instructional technology competency instrument for university lecturers in malaysia. International Journal of Academic Research in Business and Social Sciences, 8(5), 367-385. Retrieved from www.scopus.com Lecerof, A., & Paterno, F. (1998). Automatic support for usability evaluation. IEEE Transactions on Software Engineering, 24(10), 863-888. doi:10.1109/32.729686 Lee, K. H. (2005). First Course on Fuzzy Theory and Applications, Retrieved from www.scopus.com Llerena, L., Castro, J. W., & Acuña, S. T. (2018). A pilot empirical study of applying a usability technique in an open source software project. Information and Software Technology., Retrieved from www.scopus.com Llerena, L., Rodriguez, N., Castro, J. W., & Acuña, S. T. (2019). Adapting usability techniques for application in open source software: A multiple case study. Information and Software Technology, 107, 48-64. doi:10.1016/j.infsof.2018.10.011 Llerena, L., Rodriguez, N., Gomez-Abajo, P., Castro, J. W., & Acũa, S. T. (2018). Poster: "adoption of the visual brainstorming technique in the open source software development process". Paper presented at the Proceedings - International Conference on Software Engineering, 232-233. doi:10.1145/3183440.3194946 Retrieved from www.scopus.com Llerena, L., Rodríguez, N., Llerena, M., Castro, J. W., & Acuña, S. T. (2018). Applying a usability technique in the open source software development process: Experiences from the trenches. Trends in E-Learning, Retrieved from www.scopus.com Manakandan, S. K., Ismai, R., Jamil, M. R. M., & Ragunath, P. (2017). Pesticide applicators questionnaire content validation: A fuzzy delphi method. Medical Journal of Malaysia, 72(4), 228-235. Retrieved from www.scopus.com Marsan, J., Paré, G., & Wybo, M. D. (2012). Has open source software been institutionalized in organizations or not? Information and Software Technology, 54(12), 1308-1316. doi:10.1016/j.infsof.2012.07.001 Masson, A. L., Lalanne, D., & Amstutz, T. (2017). A usability refactoring process for large-scale open source projects: The ILIAS case study. Paper presented at the Conference on Human Factors in Computing Systems - Proceedings, , Part F127655 1135-1143. doi:10.1145/3027063.3053345 Retrieved from www.scopus.com Raza, A., & Capretz, L. F. (2012). Do open source software developers listen to their users? First Monday, 17(3) doi:10.5210/fm.v17i3.3640 Zhao, L., Deek, F. P., & McHugh, J. A. (2010). Exploratory inspection-a user-based learning method for improving open source software usability. Journal of Software Maintenance and Evolution, 22(8), 653-675. doi:10.1002/smr.455 |
| This material may be protected under Copyright Act which governs the making of photocopies or reproductions of copyrighted materials. You may use the digitized material for private study, scholarship, or research. |