Mobile Application Testing in Pakistan: A Survey

Research regarding MAT (Mobile Application Testing) in Pakistan is hard to discover and to the best of our knowledge, no work has been done in surveying MAT in Pakistan. In this work, we have examined the current trend and status of MAT in Pakistan. Main objective was to investigate to what extent MAT is currently applied in Pakistan software companies and what experience the companies have with using MAT. Furthermore, efforts were made to find out what testers think about MAT, e


INTRODUCTION
T he future of computing is moving towards mobile devices with rapid advancements in ICT (Information & Communication Technology).
The applications which are running on mobile devices are known as mobile applications.Mobile applications are becoming very famous as these are showing revolution in the field of information technology.Initially, mobile applications were developed mostly in the entertainment sector; however, they are now also available in other domains such as media, games, retail,


It expresses the quality or accurate behavior.


It identifies and fixes the problem.
Mobile testing refers to different types of testing, such as native and Web mobile application testing and testing of mobile devices [6].We use MAT to refer to "testing activities for mobile applications (native, hybrid and Web applications) on mobile devices with software test methods and tools in order to ensure the correctness of functionality, behaviors, quality of service, usability, security and privacy of mobile applications".Native mobile applications are coded in specific programming language (Java for Android platform and Object C for iOS) and executed on mobile devices.Native applications depend on device features, such as camera, contact list, GPS (Global Positioning System) and so on.Web mobile applications are run by a browser and generally written in HTML5 (Hyper Text Markup Language).It consists of an application server and client software.Users can access application through client software executed over browsers.Hybrid application is the combination of both the native and web applications.Hybrid is basically a web application that is built using HTML5 and JavaScript.
It is then wrapped in a thin native container that offers access to device features.
MAT is currently applied in many countries.However, actual facts and figures regarding MAT in Pakistan software companies are not collected efficiently.In this work, we examined the current trend and status of MAT in Pakistan software companies.Our main focus is on the following research question: How to optimize the performance of MAT in Pakistani software companies?
The aim is to explore the benefits of MAT as well as to improve the quality and promote MAT in Pakistan.
Objective is to examine that how much MAT is performed in Pakistani software companies and what advantages the companies have of using MAT.We have also examined affiliation among the roles, professional experience, history (age), location and size of the company.
We have collected the data from software companies located in different cities including ISD (Islamabad), SGD (Sargodha), LHR (Lahore), FSD (Faisalabad), Multan and Bahawalpur.For this purpose, a comprehensive online survey is conducted to collect responses from testing related persons.Rest of the article is organized as follows: Related work is discussed in Section 2. Our methodology, survey design and execution are stated in Section 3. Data collection from the survey, demographic information obtained from survey and main findings are presented in Section 4. Discussion and survey validity is discussed in Section 5, followed by conclusion in Section 6.

RELATED WORK
In the IT industry and in academia, MAT has recently received more attention by testing professionals due to increase in the number of testing tools and testing magazines are publishing issues exclusively on MAT [7].


Approximately half of the respondents (50%) have more than five years of working experiences.


In small companies, ET has been applied relatively more often (less than 50 employees, 56%) than in large companies (32%).


Companies older than 5 years that use ET are over 50%.Approximately 76% of the respondents who use ET are members of a separate testing organization within their company.
Madhushani et.al. [12] presented a mobile application testing procedure for Sri Lankan software development companies.They have also explained main challenges in MAT from Sri Lankan perspective.A questionnaire of 10 questions was distributed among various IT individuals and software companies in Sri Lanka.According to analysis:


93% IT industrials believes that MAT is very important for the customers.


In Sri Lanka, 92% developers develop Android mobile application and 54% develop iOS based mobile application.


Considering the level of testing, most of QA(Quality Assurance) engineers or developers use performance and security testing.Furthermore, few companies develop and test same application for different operating systems and devices.


92% testers use open source tools for the testing.


Main challenges in MAT are variety of mobile devices and different operating systems.

METHODOLOGY
We From these three RQ's, we have designed SQ's.SQ's and their mapping to RQ's is shown in Appendix-A.

SURVEY DISTRIBUTION AND RESULTS
Online Survey questionnaire was served to various mobile applications developers, testers, managers and other test relevant persons that cooperate with managers or testers.
For this purpose, we choose Google Docs platform to serve and receive online replies from respondents.We have also posted the survey link through other channels such as mailing list, LinkedIn, Twitter and Facebook.We collected 56 online replies of our survey questionnaire.
The results of data collection of survey questionnaire in graphical representations are discussed next.

Demographic Information
The demographic information of survey respondents is shown in Figs.1-6.Figures in the graphs show frequencies and percentage (for example, 24, 46% means that 24 respondents of survey select that choice and its percentage is 46).Respondents' role and the total working experience of the respondent in current organization is shown in Figs.1-2 respectively.73% respondents' are either working as manager or developer in the company.Whereas, 9 out of 56 respondents' (16%) are working as testers.Most of the respondents (41 out of 56) have more than 3 years of working experience in their current role and 15 respondents (27%) have less than 3 years of working experience.Company location, size and age of the company is shown in Figs.3-5 respectively.In 56 responses, 49 respondents belong to four cities (LHR, FSD, ISD and SGD) whereas 7 respondents belong to other cities (Multan and Bahawalpur).The characteristics of the mobile applications that respondents' organization deals with are shown in Fig. 6.The most frequently mentioned mobile application characteristics is 'high performance' (30%), followed by 'high security' (23%) and 'complex GUI' (18%).

Main Findings
We used SPSS [14] tool to analyze the replies of this questionnaire.Statistical tests such as Cross-Tabulation, Fisher Exact Tests and Pearson Chi-Square Tests [5,15] have been computed to examine the results.
Statistical tests give the information that can be used to assess the significances of increases or decreases in the results.Correct selection of statistical test depends on data type as well as aim of the analysis.In case of wrong selection of statistical test, required results may not be valid and can mislead to conclusion of the research.In this section, survey results relevant to each RQ are reported.

RQ-1
What are the abilities of those who perform MAT in Pakistan and what are the benefits for their companies of MAT?
In the following subsections, results related to RQ-1 for each sub-question is reported.

FIG. 8. RESPONSES FROM RESPONDENTS REGARDING TESTING TIME OF MAT
H0: There is no difference of performing MAT with respect to time.
H1: There is a difference of performing MAT with respect to time.
To test these hypotheses, we used Pearson Chi square test.We used SPSS tool for all calculations and the value of significant level alpha (α) is set to 0.05.Table 5 shows the test results.
Table 5 shows that p= 0.891 > α (0.05), hence accept hypothesis H0 (There is no difference of performing MAT with respect to time) and reject hypothesis H1.

RQ-1.7:
What are the key advantages to the company of using MAT?For hypothesis testing of RQ-2, we performed Crosstabulation analysis and Pearson Chi-square tests in SPSS.
We set the significance level alpha (α) = 0.05.Test results are shown in Table 9.

It turns out that only hypothesis H0-1 could be rejected.
There is no difference in frequencies of performing MAT for testers.

RQ-2.7: What factors are dissimilar when testers perform
MAT among SGD, LHR, ISD and FSD?Some dissimilar factors were also found in survey research as results of SQ-1.1 and SQ-1.3 shows that 30% of respondents of ISD are managers of software companies.
There is no professional tester found in FSD in our survey.
Mostly developers and managers perform testing of applications by themselves.Results of SQ-1.3, SQ-1.4 and SQ-1.5 show that biggest and oldest companies are located in ISD.Results of SQ-1.2 show that most experienced developers and managers also belong to ISD.

RQ-3.3 and 3.4: What is the opinion of testers regarding advantages and advantages of MAT?
In RQ-3.3, and RQ-3.4,we asked respondents to give their opinions on advantages and disadvantages of MAT by selecting at most 5 options (shown in Table 12).31% respondents' believe that 'effectiveness and efficiency' are the main advantage of MAT.On the other hand, 26 % respondents' have the opinion (shown in Table 13) that MAT is 'hard to report' and is 'inflexible'.Data collection from SQ-3.5 (issues in MAT) is listed in Table 14.Out of 56 respondents, 34% believe that the main issue in MAT is its 'security'.
In RQ-3.6, we asked respondents to give their opinions in connection with the improvements in MAT by selecting 3 options.Results are listed in Table 15.20% respondents believe that 'testers may be trained' in order to improve current status of MAT in Pakistan.18% believes that 'adapt latest technology' in order to improve MAT.

DISCUSSION AND VALIDITY OF SURVEY
Results presented in previous section are discussed in this section.Furthermore, the most pressing issue of our study is also discussed.


There is no difference in performing MAT to dissimilar level of professional experience.


There is no difference of performing MAT in different cities.


There is no difference of performing MAT in different sizes of companies.


There is no difference of performing MAT in different ages of companies.


MAT performance is not affected by the existence of separate testing company. 30% of respondents of ISD are managers of software companies.There is no professional tester found in FSD in our survey.Mostly developers and managers perform testing of their applications themselves.
Regarding RQ-3, we found the following:  More than half of the respondents' (52%) has the opinion that that 'trained and professional team and skill' and 'Quality Assurance methodologies' are the basics for defining MAT.


MAT is 'effective' and 'efficient' (advantages) and it is 'hard to report' and 'inflexible' (disadvantages).


Main issue in MAT is 'security' and 38% respondents believe that 'testers may be trained' and 'adapt latest technology' in order to improve MAT.

Validity of Survey
Every empirical study has shortcomings, so does our study too.Some issues related to our study are discussed next.Unfairness Selection: Firstly, our research consists powerful pertinent.Our goal was fully concerned to the testing of mobile applications.This is why we could not approach the various respondents at initial stages.For instance, a developer or tester who is not familiar with MAT hears at a glance the survey regarding MAT; he/ she might want to ignore the survey as soon as possible.
That is the main cause of achieving 100% encouraging replies (respondents knows about MAT earlier).
Secondly, we conducted our survey mostly in four cities (SGD, LHR, ISD and FSD) but we still obtained five replies other than above mentioned cities.In the meantime, we were not successful to get maximum responses due to restricted dispersion of the survey.question and its answer must be understandable by the respondents.So, to overcome the internal validity threats of our survey, we evaluated and reorganized our survey questions many times.For this purpose, we got help from experts/professors from academia and institutions to make our survey questionnaire easy, interesting, relevant to the topic, comprehensive as well as clear for respondents.Thus, we enhanced the survey based on time to time evaluation to move up its internal strength.

Statistical Conclusion Validity:
There is a powerful link among reliability of statistical results, size and coverage of the sample.As quantity of sample increases, ultimately, broad coverage of sample also increases so statistical results becomes much valuable.
In our survey, the size of the sample is not very big that is why there is a great threat to our research on the subject of validity of statistical results.This threat can be minimized in upcoming time by gathering a great number of samples.

CONCLUSION
In this work, an online survey is conducted to investigate the current status and trends of MAT in Pakistan and we found some interesting answers on the practice of MAT.
In total, we received 56 replies from software engineers out of which 51 were from software companies located in Islamabad, Lahore, Faisalabad and Sargodha.It comes to our notice that there are few professional testers in all our dataset, this is why, most of the developers and testes even managers of software companies perform testing themselves.Results are very fruitful to know that what kind of tools, mobile platforms and the test strategies are considered as best for MAT along with advantages, disadvantages and how improvements can be made in current MAT practice in Pakistan so it can be valuable and proficient to managers, developer and testes as well as companies.
56 responses in total is not a very good statistic amount.
In future we would like to implement the survey again and try to get more responses and include other big cities in our survey such as Karachi and Peshawar.
Furthermore, in order to get a higher response rate and a more representative set of responses, we would like to have an agreement with some companies and make sure that employees of those companies answer our survey.

RQ- 1 . 5 :
What is SW engineer trend to perform different types of testing.Data collection from SQ-1.10 and SQ-2.3 shows that system testing (25 out of 56) is performed more times as compare to other types of testing.On the other hand acceptance testing is performed very less (9 out of 56).Survey results for RQ-1.5 are shown in Fig. 7. RQ-1.6:In what circumstances, SW engineer uses MAT for their company?Testing of mobile application is generally done during application development or after the development.Almost half of respondents (27 out of 56) perform MAT during the development and other half performs MAT after application development (as shown in Fig.8).In order to check whether there is any difference by applying MAT during application development or after the development, we stated following null hypothesis.

4. 2 . 3 . RQ- 3 :RQ- 3 . 1 :
What are the opinions of testers regarding MAT and how improvements can be brought in existing practice of MAT in Pakistan?What are the basics, SW engineers think as necessary for defining MAT?In RQ-3.1, we asked respondents to define MAT by selecting three choices.Result is shown in

RQ- 3 . 5 and RQ- 3 . 6 :
What are the issues in MAT and what are the key steps for improvement in existing practice of MAT in Pakistan?

External Validity:
An additional flaw of the survey is the small amount of replies.This shows a vast hazard to external soundness of the survey.More than 500 mobile applications developers or tester could be found in above mentioned four cities but we got only 56 replies.Moreover, we could not identify accurately how much percentage of all the mobile applications developers or testers of above said four cities we approached and how much is the rate of their responses.Internal Validity: Strength of survey depends on design of survey questions.If research questions are precisely converted into survey questions and every one of the possibilities is enclosed by answer choices thus the quality of survey enhanced.It means, each survey

Table 3
1.1 Which mobile platform engineer prefer for mobile application development and testing?Table 1 shows the distribution of mobile platforms that survey respondents' use for mobile application development and testing.54 out of 56 (96%) respondents' FIG. 1. RESPONDPENTS' ROLE FIG. 2. RESPONDENTS' WORKING EXPERIENCE FIG. 3. RESPONDENTS' COMPANY LOCATION FIG. 4. RESPONDENTS' COMPANY SIZE FIG. 5. AGE OF RESPONDENTS' COMPANY FIG. 6. MOBILE APPLICATION CHARACTERISTICS develop and test mobile application in Android and iOS.This is because 82% mobile users in Pakistan use Android based Mobiles and 12.9% use iOS, according to online report [13].RQ-1.2:What are the main characteristics of mobile application?Main characteristics of mobile applications according to survey respondents' are listed in Table 2. 30% respondents (17 out of 56) believe that 'high performance' is the main characteristic of their application.13 respondents (23%) said that 'high security and safety' is the main characteristics of mobile applications that they developed and test.RQ-1.3:What type of mobile applications SW engineers develop and test?

Table 6
shows that 19 out of 56 respondents (4 managers, 11 developers, 3 tester, 1 other) believes that most important benefit to the company of performing MAT is that 'the company produces high quality mobile application'.15respondents'believethat'companygetstheattention of customer' by using MAT. [RQ-2.3]H0-3:There is no difference of performing MAT in different cities. [RQ-2.4]H0-4:There is no difference of performing MAT in different sizes of companies.[RQ-2.5]H0-5:There is no difference of performing MAT in different ages of companies.[RQ-2.6]H0-6: MAT performance is not affected by the existence of a separate testing company.

TABLE 5 . PEARSON CHI SQUARE TEST RESULTSTABLE 6 . COMPANY BENEFITS BY USING MAT All
the survey respondents' were familiar with MAT.Data related to RQ 2.1 and RQ 2.2 is listed in Table7.Responses

TABLE 8 . RESPONDENT WORKS AS SEPARATE TESTERTABLE 9 . TEST RESULTS FOR RQ-2.1 TO RQ-2.6 RQ-3.2:
What uniqueness do SW engineers feel about