Methodological choices of HR research conducted in Asia

Abstract

A twenty-year study of the Human Resource (HR) practices–outcome relationship has found that more rigorous methodologies have been adopted over time. However, several problematic features such as cross-sectional, single-informant, and single-level designs continue to be adopted (Bainbridge et al., Human Resource Management, 2016). Responding to calls for increased contextualization of research by investigating the relationship between the location of data collection and the methodological choices of researchers, this study answers the question “How unique are the methodological choices of HR research conducted in Asia?” Applying content analysis to 241 published articles, we compare internal, external, construct and statistical conclusion validity of studies collected in North America (n=66), Europe (n=95) and Asia (n=80, including 57 studies from China). Results show that despite similarities in cross-sectional, single-informant and single-level designs across regions, research conducted in Asia is mainly undertaken via field studies, using subjective outcome measures at the organizational level, following a post-predictive design. In addition, studies from Asia are more recent, and show a shorter time gap between data collection and publication. Theoretical and practical implications embedded in the dynamic context of Asia in general, and China more specifically are discussed.

Introduction

Human Resource (HR) scholars and practitioners emphasize the importance of a robust methodology to study the relationship between HR practices and employee and organizational outcomes (Bainbridge, Sanders, Cogin & Lin, 2016 ; Combs, Liu, Hall & Ketchen, 2006; Guest, 2011; Hayton, Piperopoulous, & Welbourne, 2011; Paauwe, 2009; Wall & Wood, 2005; Welbourne, 2011; Wright & Gardner, 2003). Strong methodologies build confidence in research findings by providing evidence that HR practitioners can draw on to inform decision making (see Bainbridge, et al. 2016; Rynes & Bartunek, 2017). In addition, sound methodologies contribute to building a rigorous body of knowledge and help to solve challenging organizational problems.

Despite the acknowledged importance of strong research methodologies, HR scholars have questioned if the methodologies employed in the HR field provide a sufficient basis for drawing valid conclusions (Wright & Ulrich, 2017). For instance, the prevalence of common method bias where a single source of data measures both HR practices and outcomes, such as firm performance, has been criticized (Wright, Gardner, Moynihan & Allen, 2005). This kind of research design leads to an overestimation of the relationship between HR practices and outcomes (Gerhart, Wright & McMahan, 2000; Huselid & Becker, 2000, Sanders, & Frenkel, 2011). In addition, the pervasiveness of cross-sectional research designs, where HR practices and performance are measured at the same time has been questioned. Such an approach is a more accurate predictor of past performance instead of future performance and prevents valid causal conclusions (Huselid & Becker, 2000; Wright et al., 2005; Wall & Wood, 2005). Single-level research designs are also problematic as they do not account for variance apportioned at different levels, thwarting the supposition that HR practices enhance employee and/or organizational outcomes (see Guest, 2011; Paauwe, 2009).

Bainbridge et al. (2016) content analysis of twenty years of studies investigating the HR practices – outcome relationship found significant improvements across many validity-related methodological choices over time. For instance, multi-level research designs are increasing, and there has been a decrease in post-predictive research designs, in which performance is measured prior to HR practices being measured. There has also been a reduction in direct effect testing. The growing trend of mediator and moderator research to explain the relationship between HR practices and outcomes and uncover factors that influence this association, as well as less usage of same source data to measure the HR and outcome variables are also positive changes. Notwithstanding this, problematic designs which include cross-sectional, single source and single level features are common.

Several scholars have called for increased attention to the context in which research is carried out (Barney & Zhang, 2009; Li, Leung, Chen & Luo, 2012; Tsui, 2006, 2013; Sheldon & Sanders, 2016). For instance, John (2006) calls for stronger engagement with the context of the data collection. In terms of geographic context, scholars have emphasized the uniqueness of Asia’s social, cultural, historical, and political mosaic as well as economic and market reforms, in order to highlight the importance of context (Tsui, Wang & Xin, 2006). Related to China specifically, Tsui (2006, 2013) emphasizes the importance of ‘inside out’ approaches that are sensitive to knowledge about the environment in contrast to ‘outside in’ research in which well-known topics examine ‘how they are manifested’ in the Chinese context (Tsui, 2006: 3; see also Sheldon & Sanders, 2016). Although these calls emphasize the importance of context in relation to theoretical questions, the research setting may also influence the suitability of different methodological treatments. In qualitative research, it is well established that context can shape methodology (Townsend, Loudoun & Lewin, 2016). Collaboration research between Chinese and UK scholars indeed reveal different ontological and epistemological assumptions and research approaches between the two teams (Easterby-Smith & Malina, 1999). However, in quantitative research less attention has been given to this issue.

As the Asian and Chinese economy in particular, has gained momentum in the past few decades, increased research has been conducted in this setting (Bainbridge et al., 2016). This is not surprising given claims that management research conducted in Asia could potentially contribute to global management knowledge (Meyer, 2006). An important question then is: To what extent does research with an Asian or Chinese data sample utilize different research designs compared to other geographic settings? Understanding the influence of research settings on methodological choices would raise researchers’ awareness of potential opportunities and constraints of conducting research in different geographical settings. By focusing the spotlight on Asia, such an investigation would enable scholars doing work in this region to strengthen the rigor of their research. As such, this study examines how methodological choices differ across regions. We do this by undertaking a content analysis of 241 studies between 1995 and 2014 using samples from Asia, Europe and North America.¹ We begin by presenting our research methodology before examining the internal, external, construct and statistical conclusion validity of studies using data collected from these regions. We then discuss the theoretical and practical implications of our results.

Method

Sample and procedure

This research utilizes the dataset developed by Bainbridge, et al. (2016) who examine the methodological choices of researchers studying the HR practices – outcome relationship in six leading management and six leading HRM journals between 1995 and 2014: the Academy of Management Journal (AMJ), Human Relations (HR), Journal of Applied Psychology (JAP), Journal of International Business Studies (JIBS), Journal of Management Studies (JMS), and the Strategic Management Journal (SMJ), British Journal of Industrial Relations (BJIR), Human Resource Management (HRM), Human Resource Management Journal (HRMJ), Industrial & Labor Relations Review (ILLR), International Journal of Human Resource Management (IJHRM), and Personnel Psychology (PP).² These journals were selected because their studies are widely cited, making articles published in these outlets disproportionately influential on the field. In this follow-up study, we compare the methodological choices of studies undertaken in North America (n = 66), Europe (n = 95) and Asia (n = 80, including 57 from China)³ across four aspects of validity: internal, external, construct and statistical conclusion validity. The 57 studies from China came from Chinese mainland (39 studies), Hong Kong (2 studies) and Taiwan (16 studies). Although Taiwan’s social and economic systems differ from Chinese mainland, we did not find significant differences between the validity elements of studies conducted in Taiwan and Chinese mainland. Because of this and the small number of studies in Hong Kong, we collapsed the studies together in our analysis.

Two criteria were used for inclusion in the sample: i) each study must test the effect of multiple HR practices on an outcome, and ii) the study must contain empirical quantitative research. Multiple HR practices are defined as the presence of two or more substantively different HR practices. Outcomes are defined as measures of economic performance (e.g., profitability, return on assets, return on equity), operational performance that focuses on outputs (e.g., productivity, production costs, quality, sales), and non-economic attitudinal and behavioural outcomes (e.g., job satisfaction, organizational commitment, turnover intentions, organizational citizenship behaviours) (see Bainbridge et al. 2016).

We undertook a content analysis of all studies using a hierarchical system of codes based on Aguinis, Pierce, Bosco, and Muslin’s (2009) taxonomy of methodological choices across four aspects of validity (internal, external, construct, statistical conclusion). Data was coded into 61 dichotomous variables using dummy coding (0 = methodological choice not made in the study; 1 = methodological choice made in the study) and four continuous variables. Methodological choices relevant to each aspect of validity are outlined in the following material.

Results

Region and journal outlets

We found significant differences in publication patterns by journal outlet and region (χ2 (22) = 141.40, p < .01). As displayed in Table 1, studies that use data collected in North America were more often published in AMJ, HRM and ILLR. European studies were often published in IJHRM and HRMJ and studies employing samples from Asia and Europe were most often published in IJHRM. Although studies conducted in China were more likely to be published in AMJ and less in IJHRM than studies conducted in Asia, these differences were not significant (χ2 (11) = 8.21, n.s.). Over the twenty year timeframe, the results show that HR studies were most likely to be conducted in Europe (38%), and Asia (32%), while only a quarter of the studies were undertaken in North America (26%). The results also show that studies conducted using European research samples remained stable over the 20-year timeframe, in comparison to the number of Asian studies which dramatically increased, especially studies conducted in China. Research undertaken in North America saw a significant decrease over time.

Table 1 Journal Outlet by Region

In the following section, we report differences across regions. Table 2 summarizes the significant results across four aspects of validity for the three regions.

Table 2 Descriptive Statistics Where Significant Differences were Found: Internal, external, construct and statistical conclusion validity

Internal validity

Internal validity concerns issues of causality and the accuracy of conclusions drawn regarding whether statistical relationships found between variables imply cause (Cook & Campbell, 1976). A high degree of internal validity provides convincing evidence of causality; likewise if a study has low internal validity, then little or no evidence of causality is provided. Three conditions must be satisfied to claim that a relationship between X (the presumed cause, in this case, HR practices) and Y (the presumed outcome), is causal: i) presence of a relationship between HR and an outcome; ii) time precedence (i.e., first HR practices, followed by an outcome); and iii) is non-spuriousness (e.g., excellent line managers effective in implementing HR practices drive superior outcomes, or a positive economic situation which explains both HR practices and the outcome) (Cook & Campbell, 1976).

In this study, we assessed internal validity through an examination of the following methodological choices: type of data (field vs archival data), data structure (cross-sectional, longitudinal), predictive design (post-predictive, retrospective, contemporaneous, predictive; Wright et al., 2005), and the form of the HR practices–outcome relationship (direct, mediation, moderation, moderated mediation).

We found that regions differed in how the research was conducted against the inferences made. Field studies (as opposed to archival research) were more prevalent in Asia (96%; China: 94%) and Europe (82%) than in North America (75%) (χ2 (2) = 28.33, p < .001). In addition, we found significant differences in the predictive design of studies (χ2 (4) = 19.21, p < .001). In Asia, a post-predictive design, in which the performance measurement took place before the HR practices were measured was most common (55%; the same for China alone). The most notable consequence of this is that any inference of causality for these research designs is not valid (Wright et al., 2005). In Europe, most studies (45%) employed a contemporaneous predictive design by measuring HR practices and the outcome at the same time. North America most often utilized predictive design (32%) which assess if HR practices at time 0 are related to an outcome measure at a subsequent time. This research design allows for stronger conclusions about the causal relationship between variables.

No significant differences in research design by region were found pertaining to cross sectional studies (χ2 (2) = 1.08, n.s.), which was by far the most prevalent design (93%). Only modest numbers of longitudinal studies (7%) were conducted during the twenty-year time span. The extent of cross sectional studies was alarming given extensive criticism of this approach and the inability to provide evidence of a causal relationship between HR practices and an outcome (Wright et al., 2005).

The most common choice for examining the HR practices – outcome relationship was a mediation model (36%). This was followed by moderation (31%), direct effect (26%), and moderated mediation (7%) models. We found no significant differences across regions for such choices (χ2 (6) = .31, n.s.).

External validity

External validity concerns the extent to which study findings can be generalized across time, location, setting, and entities (Cook & Campbell, 1976). External validity was assessed through an examination of the level of analysis of the dependent variable (individual, team / workplace / business unit, or firm level), outcome measures (subjective and/or objective), level of measurement (single versus multi-level), industries (single versus multiple industries), response rates, year of data collection compared with the year of publication and evidence of common method bias.

We found differences pertaining to the level of measurement across regions (χ2 (4) = 24.03; p < .001). Most studies were conducted at the firm level (Asia 55% (China: 46%), Europe 45% and North America 40%); however, research conceptualized in Europe and North America also showed higher levels of research undertaken at the employee level (40% and 21% respectively). Over the twenty-year period, there has been minimal investigation of how HR practices affect team outcomes or meso-level research in Europe (15%) and Asia (10%; China: 19%), in comparison to studies conducted in North America (39%).

Significant differences were identified regarding whether studies undertook investigations in single or multiple industries (χ2 (2) = 4.27; p < .05). Asia (52% (China: 38%)) and Europe (59%) were more likely to include multi-industry samples, while the North America largely focused on single industry investigations (67%). Response rates differed across regions (F (2194) = 4.13, p < .05), with studies undertaken in Asia securing higher response rates (51%) compared to those in Europe (38%) and North America (45%). We also found significant differences pertaining to the year of data collection (F (2144) = 13.78, p < .01) and publication year (F (2238) = 15.33, p < .001). During the period of our review, data collected in Asia was on average gathered in 2004 and the research was published in 2009. European data was collected on average in 2001 and published in 2006. Consistent with the reduction of studies over time coming out of North America, data was gathered on average in 1999 and published in 2005. These findings illustrate a shorter time lag between data collection and publication year for Asia (4.52 years; China: 4.68 years) in comparison to Europe (5.00 years) and North America (5.75 years) (F (2, 132) =2.57, p < .10).

We found no geographic differences in relation to the usage of same source data for the independent and dependent variables (χ2 (2) =3.01, n.s.). On average 62% of the studies in our sample used same source data for the independent and dependent variables. In addition, there were no regional differences in the informants for the HR measure (78% used a single informant; F (2238) = 1.44, n.s.), the informants for outcome measure (67% used a single informant; F (2230) = .60, n.s.), or the levels in the model being tested (84% of research designs contain only one level). There were also no differences between the regions in the type of informants: 32% asked HR managers, 32% asked managers, and 36% asked employees to collect information about the HR practices and/or the outcomes.

Construct validity

Construct validity is the degree to which a questionnaire or scale measures what it claims to be measuring. It concerns the fit between the measure and the underlying constructs they are designed to measure (Cook & Campbell, 1976). Construct validity was assessed through an evaluation of the number of informants for the HR measure as well as the number of informants for the outcome measure and the number of controls in place.

We found differences across regions pertaining to whether scholars used objective (e.g., productivity, profit, or return on assets, from externally recorded and audited accounts) or subjective (e.g., asking respondents their perceptions of company performance) outcome measures (χ2 (4) = 30.21, p < .001). Studies conducted in Asia (79%) and Europe (71%) were much more likely to contain subjective outcome measures than scholars who used data from North America (44%). Very few studies in Asia (14%; China: 18%) and Europe (17%) drew on objective outcome measures compared with those in North America (41%). There were also differences in the number of variables used in research models (F (2238) = 4.74, p < .01). Studies utilizing European (6.72) and North American (6.26) samples employed more variables than Asian ones (4.60). There were no differences in the number of controls in the models (mean = 4.39; F (2232) = .98, n.s.) used in the study.

Statistical conclusion validity

Statistical conclusion validity concerns the ability to make inferences about relationships between variables (e.g., HR practices and outcomes) based on statistical evidence in relation to co-variation and prediction (Cook & Campbell, 1976; Scandura & Williams, 2000). Statistical conclusion validity was assessed through an examination of sample size and data analysis technique.

We found significant regional differences pertaining to sample size (F (2238) = 3.68; p < .05). Asian studies were more likely to employ smaller samples (n = 259), compared with those in North America (n = 1240) and Europe (n = 1328). There were no differences in data analysis technique by regions (χ2 (12) = 1.29, n.s.) with regression (65%) using the dominant method. AN[C]OVA and MAN[C]OVA (6%); correlation (4%); multi-level analysis (7%); structural equation modelling [SEM], path analysis (21%); other quantitative methods (3%) were less prevalent.

Comparison of Asian studies across outlets

Consistent with our specific interest in studies conducted in Asia, we examined the methodological features of Asian studies published in different outlets. More specifically, we compared the studies published in the Academy of Management Journal, (AMJ) and Human Resource Management (HRM) with those published in the International Journal of Human Resource Management (IJHRM). This analysis is meant to answer the question of “why data collected in Asia, including China, are more likely to be published in IJHRM instead of AMJ and HRM”. We found that Asian HRM studies, published in IJHRM in comparison with AMJ and HRM are characterized as being more single-level as opposed to multi-level (87% vs 65%), examining more direct relationships between HR and outcomes (76% vs 62%) as opposed to moderation (77% vs 50%) and mediation relationships (75% vs 64%), using more subjective data (84% vs 68%) as opposed to objective data (12% vs 20%), and having a more international focus (42% vs 25%).

Discussion & Conclusion

This study examines the methodological choices of research investigating the HR practices – performance link by geographic region over a 20-year period. We focused on internal, external, construct and statistical conclusion validity of 241 studies using data collected from Asia, Europe and North America. Some of our results are universal phenomena and not limited in relevance to Asia, although in line with our aim we present implications related to the HR field in the context of Asia.

The last two decades show an increase in studies using data collected in Asia, particularly China, and a reduction in studies out of North America (Bainbridge et al. 2016). This is not surprising given the far-reaching institutional change and societal transformation occurring in China. Regarding the internal validity, the results show that the three regions did not differ in terms of the prevalence of cross sectional research designs and the form of the HR practices -outcome relationship (direct, moderator, mediator or moderated mediator). The results found different sources of data with studies coming out of North America tending to utilize more archival data while Europe and Asia, including China, were more likely to use field study data. In fact, 96% of studies from Asia and 94% of research undertaken in China came from field studies. The three regions also diverged in terms of predictive design: most of the studies in which data was collected from Asia, including China, utilized a post-predictive design, where performance data collected takes place prior to HR practices data. Data collected in North America more often adopted a predictive research design in which HR practices data is collected at time 0 and performance data is collected at time 1, allowing the testing of causality. Studies that used a European sample are characterized by a contemporaneous research design in which the HR practices and performance data take place at the same time.

There were many differences by region pertaining to external validity. Studies out of North America more often came from one industry, while European and Asian studies were more likely to draw on data from two or more industries. Regarding the levels of measurement of the dependent variable, we found that data collected in Asia was typically at the organization level with higher response rates while European studies were at the employee level. In comparison, studies using data from North America were more often designed at the organization and workplace/business unit/team level. The time between data collection and publication year was shorter for studies out of Asia, compared with North America and Europe.

Our overall conclusion regarding construct validity is that there are few differences by region. Studies using data from Asia, Europe and North America mainly used single (as opposed to multiple) informants to measure HR practices and the outcome variable. The three regions varied equally on the type of informant (i.e., managers, HR practitioners and employees). We found no differences in the number of controls used in the model being tested (average four) and prevalence of same source data for the HR and outcome measures, suggesting common method bias remains a problematic feature of HR scholarship. The regions differed in terms of outcome measures (subjective versus objective performance data).

While studies that used data from Europe or Asia primarily utilized subjective data, objective performance data or a combination of subjective and objective performance data was much more common in studies that used data from North America. In addition, the number of variables in the model was higher for studies using data from a European country or North America in comparison to studies out of Asia, including China.

In terms of statistical conclusion validity, studies in Asia drew on smaller sample sizes than those conducted in Europe or North America, although, as noted above, had higher response rates. There were no differences pertaining to how data was analysed with regression the common technique across regions.

Theoretical and practical implications

There are several theoretical and practical implications of our results. The institutional theory claims that actors’ decisions are not only the result of rational decision-making seeking to improve effectiveness, but are also influenced by the institutional context in which they operate (Kostova, 1999). Following this logic, we see an opportunity to study research methods from an institutional perspective. Scholars exist in conditions of institutional duality, in that they experience pressures to obtain both internal legitimacy in the context which research is conducted (Asia in our case) and external legitimacy from the academic field as evidenced by publication in a top journal.

We suggest there is possible tension as scholars endeavour to attain internal legitimacy and respond to calls to adapt theories to explain Asia-specific phenomena or develop new models and theories relevant to the Asian context (Tsui, 2006) while at the same time needing to meet an equally pressing requirement of external legitimacy by adopting norms or assumptions around academic rigor. Our findings show that research undertaken in Asia, Europe and North America has converged to measure constructs and analyse data in similar ways. This may be due to the influence of international research teams or editor and reviewer’s preferences and demands (see Lin & Sanders, 2014). This result is evident in that validity related to the fit between measures and the underlying constructs they are designed to represent (construct validity) and the ability to make recommendations regarding the relationship between variables based on statistical evidence about co-variation and prediction (statistical conclusion validity) do not differ by geographic region. It appears that in this regard, research conducted in Asia is largely a direct application of Western practices. As the current positivist research paradigm originates from the West, established criteria in relation to what is deemed valid research constitutes isomorphic pressure which Asian scholars need to follow, should they wish to gain legitimacy in the field and publish in top journals.

Despite the isomorphic pressure, substantial variations across regions were found regarding internal and external validity, especially in relation to studies undertaken in Asia. We believe that the broader context (i.e., local institutions) of Asia, and China more specifically influences these findings. Many scholars refer to the Chinese environment as dynamic and subject to ongoing change (Krug & Hendrischke, 2008; Li, X, & Freeman, 2015; Zhang & Peck, 2016; Sheldon & Sanders, 2016). Given the pace of change across Asia, it is not surprising that field studies which enable the collection of real-time data are the dominant methods for undertaking research in this region. The prevalence of field studies is also a result of the lack of reliable statistical systems in developing countries such as China (Fang, 2004). Oftentimes researchers are left with no other choices but to rely on field studies and subjective measures, because more objective archival data is not available. In the Asian culture, field data collection is often dependent on researchers’ personal contacts (i.e., guanxi) and convenience sampling (e.g., Sun, Aryee, & Law, 2007; Takeuchi, Chen, & Lepak, 2009), which leads to smaller sample sizes and higher response rates in this area. Convenience sampling makes it challenging to focus on a single industry, because more often than not the researchers’ contacts are from multiple industries, whereas the available sample size in a single industry is severely constricted.

These features of the Asian context present barriers for researchers to conduct high-quality research in this area. In the dynamic context of Asia, where research is predominantly undertaken through field work utilizing a cross sectional research design, there is risk that the research context and results could vary from one day to the next. This means that researchers must be careful as to the best timing of their data collection. For example, Shaw and Shi (2017) mention that Chinese organizations face unique temporal and seasonal challenges addressing mass employee turnover (e.g., after the Lunar Year Holiday and after bonus payments). This has implications for researchers’ decisions regarding when they should enter the research site and whether a predictive or longitudinal research design is feasible. It also requires scholars to consider the influence of their temporal decisions on their research findings.

The overreliance on subjective measures risks undermining research validity. While research has found that subjective measures of firm performance are equivalent to those used for objective performance (Wall, Michie, Patterson, Wood, Sheehan, Clegg & West, 2004), simultaneous use of subjective measures of both HR and performance could compound the problem due to common method bias (Podsakoff, MacKenzie, Lee, & Podsakoff, 2003). Given the pervasive use of common source data in HR research, scholars may like to reconsider the current convention of relying on subjective measures of performance and shift towards an objective or combined approach. This is especially important for conducting research at the employee level, where it has been shown that subjective measures should not be used as proxies for objective measures and scholars are cautioned about using the terms interchangeably (Bommer, Johnson, Rich, Podsakoff & MacKenzie, 1995).

Despite the challenges, the unique environment in Asia also offers opportunities to researchers. The many changes in this area provide scholars with a fertile research setting for studying the effects of HR on employees, teams, and organization performance over time. For instance, related to the HR process approach (Bowen & Ostroff, 2004; Sanders, Shipton & Gomes, 2014; Ostroff & Bowen, 2016) in general and attribution theory (Nishii, Lepak & Schneider, 2008) more specifically, scholars could investigate how people attribute the locus of causality (i.e., the internal versus external dimension of attribution): To what or whom do employees attribute organizational change? For example, do employees attribute the driver of change to their employer (internal) or government (external to the organization)? How do they respond to the change and their attribution of the change? We believe that China is an ideal context for longitudinal research for these types of questions. Further, the transition from traditional to modern management philosophies and styles and their co-existence in the country makes it a great setting for quasi-experimental studies (e.g., Yan, Peng, & Francesco, 2011), which have been underutilized in HR research. Scholars may like to make better use of these opportunities.

The challenges we have identified also point to directions moving forward. We have acknowledged that the heavy reliance on field studies and subjective measures may be due to existing databases not capturing the dynamism of the Asian or Chinese context or because they are incomplete. We recommend that academic institutions such as the Chinese Academy of Social Sciences and the Chinese government start building databases such as the Workplace Employee Relations Surveys (WERS) in the United Kingdom (UK) or the Household, Income and Labour Dynamics in Australia (HILDA). The benefit of archival data is that trends over time can be depicted, and databases are generally of good quality in terms of construct validity. For example, HILDA provides longitudinal data on the lives of Australian residents against a wide range of factors such as family dynamics, economics, well-being and the labour market. These databases are administrated by government agencies; for instance, the Department of Business, Innovation and Skills in the UK is responsible for collecting WERS data, which are subsidized by the government, and open to scholars. Support of trade associations should also be garnered to allow researchers to access organizations in specific industries and conduct in-depth analysis. Shaw and Shi (2017) point out that turnover research in Western industries has greatly benefited from this. We observe that some joint action in data collection has been taken. For example, the study of Shen, Au, and Birtch (2016) utilize a representative sample collected in remarkable joint efforts of multiple government departments and a research association. Such progress is welcome.

Furthermore, changes are needed from Chinese universities and business schools. We believe that the shorter time lag between data collection and publication in China is linked to the Chinese academic system. The Chinese academic system for performance and promotion is similar to the US system (Jia, You and Du, 2012; Tsui, 2006; Van de Ven & Jing, 2012). Scholars, especially early career researchers (ECR) endeavour to publish as soon as possible to meet promotion criteria. In many universities, quantity is gauged over quality and impact (Tsui, 2013). We question whether this practice is the best strategy given the time it takes for a rigorous theoretical review and the collection of high-quality data. In recent years, there have been numerous calls for Chinese researchers to conduct indigenous research (Leung, 2012; Li, et al., 2012), develop a Chinese theory of management (Barney & Zhang, 2008; Sheldon & Sanders, 2017) and live up to the idea of socially responsible scholarship (Tsui, 2013). Concurring with others, we encourage Chinese Business Schools to rethink their academic performance evaluation system and take a long-term perspective by considering the value of a scholarship to Chinese society. Such changes would encourage researchers to conduct socially responsible research (Tsui, 2013) and direct their attention to addressing grand societal challenges (George, Howard-Grenville, Joshi, & Tihanyi, 2016). This may also assist in managing the challenges associated with institutional duality in that research which is conducted in Asia and trapped between the paradoxical aims of local relevance and international publication requirements (Meyer, 2006).

Finally, we believe the entire management scholarly community needs to adjust its practices. Referring to evidence-based management and the quality of data, Rynes and Bartunek (2017) warn that journals are more likely to publish papers which report significant results, instead of non-statistically significant but important findings. We believe that some research designs such as single level, single informant and same source research designs often lead to more significant results because the data is not independent of each other. We question whether (associate) editors, reviewers and researchers receive sufficient incentives and are motivated to design robust and valid research (see also Lin & Sanders, 2014).

Limitations

Sheldon and Sanders (2016) emphasize that variation within China is immense and undertaking research at the country level is inappropriate because of the differences in regions and industries. One of the limitations of this study is that our data did not allow us to examine the variance of methodological choices within China. In the future, scholars should consider providing more detailed information pertaining to regions within a country with regards to data collection.

It is important to note that our study considers data collected in Europe, North America, or Asia (China). This does not necessarily mean that the authors are from these locations. According to our observation, it is more likely that at least one of the authors is from the country in which the data is collected. The practice of collecting data from Asia is likely to have emerged in the early 1990s when Chinese people became more mobile, leading to the growth in the number of China-based multinational enterprises (MNEs) and Chinese academics who had studied and worked abroad. In addition, such mobility has produced opportunities for Western scholars to undertake research in Asia by supervising international research students (Sheldon & Sanders, 2016) and collaborating with Chinese scholars. A question we have not answered is whether Chinese or Asian scholars were trained at a western university, or one of the approximately 2000 Chinese universities. We also did not consider domestic scholars that publish in their own language. These studies are largely invisible to Western scholars (Bainbridge et al. 2016; Sheldon & Sanders, 2016).

In sum, despite these unanswered questions and similarities across regions pertaining to cross-sectional, single-informant and single-level designs, research conducted in Asia is distinguished by several features. Studies that utilize data collected from Asia mainly employ subjective outcome measures at the organizational level gathered via field research following a post-predictive design. In addition, studies from Asia are more recent, and show a shorter time gap between data collection and publication, have smaller sample sizes but higher response rates.