Research Trends -

The first APAC Research Intelligence Conference was attended by 109 people from 70 institutions, coming from 8 different countries worldwide. The topic of discussion at this two day event hosted at The Nanyang Executive Center at NTU in Singapore was Research Excellence, the challenges which institutions face with regard to managing research and the best practices employed to optimize research strategy and impact. The idea to organize this event stemmed from a common interest in having a platform to facilitate open discussion on the topic by dedicated professionals, and that was certainly achieved as 9 speakers took the stage to share their insight and experiences. This article reviews selected parts of each speaker’s presentation.

Group photo taken during the conference, with presenters mentioned in the article in bold.
Back row from the left – Hiroshi Fukunari, Marcel Vonder, Thomas Thayer, Anders Karlsson, William Gunn, Kevin Carlsten, Lim Kok Keng
Front row from the left - Byoung Yoon Kim, Hirofumi Seike, Douglas Robertson, Giles Carden, Michael Khor.

 

Day 1

Professor Bertil Andersson, President of Nanyang Technological University (NTU) in Singapore, presented ‘Nanyang Technological University, Singapore: A Drive in Excellence’.

Professor Andersson described Singapore as a country with a vibrant eco-system of world-class research producing institutions. He highlighted the important role of the Singapore government, as not only talking about developing the knowledge economy, but also walking the talk by providing funding, having a dedicated Research Innovation & Enterprise Council chaired by the Prime Minister, using 5-year planning cycles, and by having a tradition of philanthropic endowments and incentivizing private donations. NTU is one of the fast-rising universities in both the world ranking and research impact. Figure 1 shows NTU’s Field-Weighted Citation Impact surpassing that of Asia’s top institutes by 2012.

Professor Andersson attributed their success to being young and having been able to start from scratch rather than reorganizing an existing structure, receiving long-term generous finance, and being able to recruit senior and junior faculty from abroad by maintaining a strong international profile.

 

“A university is about its people, its people and its people……its good people. I think personally the biggest secret to our success has been that we’ve been able to recruit top people from Europe, United States and Asia… of a very high caliber. And we also recruited many young people. The superstars of tomorrow have come to NTU in big numbers and we had funding for that”.

Professor Bertil Andersson, NTU

 

 

 

Figure 1 –The field-weighted citation impact of NTU and selected comparator institutions 2004-2012. Source: SciVal.

 

Professor Byoung Yoon Kim, Vice President of Research at the Advanced Institute of Science and Technology (KAIST) in South Korea presented ‘Strategic Role of KAIST in Advancing Korean Economic Development’.

Professor Kim outlined the role KAIST has played in developing Korea’s economy in the last 40 years and spoke about the role they hope to play in the next 40 years. KAIST was established in 1971 with a mission to produce professionals to transform Korea into an industrialized nation. As an initiative for change and development, it was not only a new university, but was also under a different ministry, and therefore did not share budget with the other universities. KAIST recruited the best professors worldwide and successfully contributed to Korea’s economic growth by fostering talents who established companies now known worldwide, which generate the majority of Korean’s income.

Looking forward, Professor Kim spoke about the Startup-KAIST movement, which aims to establish a model that the country should follow by spreading a culture of entrepreneurship, to develop an eco-system to help establish and globalizecompany activities. Professor Kim echoed Professor Andersson in attributing the success of KAIST in part to having started as an independent university rather than changing an existing system and culture. He said if the same money went to another existing University, it would not have produced the same results. KAIST represented a departure from the old system.

 

“KAIST has to also find out what it should be doing for the next 40 years in order to be different and justify its existence.  We should not compare our university with SNU… it has a different mission. Although my president (laughs) and most government officers are very interested in university rankings, I try not to talk about it, because it is important in a sense, but it should not be the goal…”

Professor Byoung Yoon Kim, KAIST

 

 

 

Dr. Anders Karlsson, Vice President Global Academic Relations APAC, Elsevier, presented ‘The Global Trends on Internationalization and Assessing Impact Beyond Research’.

Dr. Karlsson posed a number of questions; the most central being ‘is collaborative work better?’  He showed the positive correlation between the international collaboration share of a country and their Field-Weighted Citation Impact, found in the report prepared by Elsevier for the Department of Business Innovation and Skills (BIS) in the UK (1), and was quick to point out that correlation does not explain causality. From the same study, he presented data thatshowsthe UK’s international collaborative papers were cited 60% more often than papers collaborated on only within UK. That data was positioned as strong evidence demonstrating the leverage the UK gets from collaborating internationally, in terms of the positive effect on overall scholarly influence.

Dr. Karlsson investigated whether international articles are judged better in peer review. He used evidence provided from a study (2) which looked at papers submitted in Italy for peer review, and found that papers with more authors were judged higher in excellence.

 

“If you collaborate more, your citation impact increases, basically you have a broader base, and you reach out more broadly…

International collaboration should be high on the strategic agenda of countries which want to increase their citation impact.

Dr. Anders Karlsson, Elsevier

 

 

 

 

Dr. Giles Carden, Director, Strategic Planning and Analytics, Warwick University presented ‘Research Planning: Embedding analytics in a new research performance challenge process at the University of Warwick’.

Dr. Carden introduced Warwick University’s approach to using analytics for managing research performance, and explainedtheir imperative strategic rationales, achievements, and future direction. The context for developing analytics was to supportWarwick’s goal of becoming an undisputed world leader in research and scholarship, plus the fact that the UK’s national research exercise in part based their assessment on these types of analytics. Distribution of UK’s 1.6 billion pounds in block grants coming from the government is informed by the assessment outcome of the Research Excellence Framework (REF). Thus, Warwick developed an analytics and planning process in tandem and embedded the analytics into the process to be successful in this very competitive environment.

Dr. Carden shared analytics showing Warwick’s collaboration with the USA (see Figure 2), stating this was important to boost citation impact. He also revealed that Warwick’s Research Assessment and Planning group reviews the performance of each individual academic in a substantial post, and showed an author profile in SciVal (see Figure 2). Communication was the key to the project’s success.It was not about being out to get people, but to identify patterns that can help researchers turn their performance around. As a result, Warwick University improved academic staff accountability, grew in research income and research students, published more in high impact journals, and increased citations – along with a cultural shift within the University. In closing, Dr. Carden discussed the future of analytics and big data as likely involving predictive analytics, and also highlighted the limitations of analytics.

 

Figure 2 – The collaboration map shows the institutions which Warwick University has collaborated with represented by a bubble which shows the number of co-authored publications (2011-2013). Source: SciVal.

 

 

Figure 3 – Author profile showing the publications, citations, citations per paper and h-index of a specific author. Source: SciVal.

 

Dr. Douglas Robertson, Director of Research Services Division, The Australian National University (ANU), presented ‘The Changing World of Research Support and the Challenges of Impact from Basic Science: Some Reflections’.

Dr. Robertson has been active in research administration since 1983, and reflected on the changing nature in university research support and on some concerns. Research administration has become much more complex, and he questioned whether the quality of research is any better as a consequence. He encouraged contemplation about whether the development and current practice of research administration is really to the benefit of science and society.

“Life was very simple in 1983. When you were sent a research award, it ran to one side of A4 that said ‘we’d like to give you some money, will you please write back and say whether you’d like it. And if you could tell us what you did in three years’ time, we’d be very grateful.’ Now research contracts in the UK can run to 90 or 100 pages, of very closely typed script…there has been quite a lot of change… “

Dr. Douglas Robertson, ANU

 

 

He asked whether we are spending too much money on administrating research and not enough money on actually doing it. He also stated that several Nobel Prize winners have questioned whether they would have been funded under the current systems. Today, researchers have to report more often, get more permission, andjustify more why their research is worth investing in, while the focus is now more on the societal impact than the impact on research and other researchers.

Dr. Robertson also questioned whether the race to publish is a good thing, citing a number of studies which report observed lack of reproducibility, including one in the pharmaceutical industry where it was revealed that in only ~20–25% of the projects, were the relevant published data completely in line with our in-house findings (3).

“I find it challenging to figure out how we create an effective research environment rather than one that is easy to measure. I am of the opinion that if you are using public money, and produce work that cannot be reproduced, it is not a good outcome, the aim is that you publish so that others can build on your publication, that you patent so that others can build on your invention, and if your publication does not achieve that, we have concerns. Particularly in the life sciences, the pressure on scientists is phenomenal…”

Dr. Douglas Robertson, ANU

Finally, Dr. Robertson raised the importance of curiosity driven research and concerns about the increased shift in focus to applied science. Scientists are increasingly required to indicate what their research will be used for rather than being left to freely explore the unknown. He underlined the importance of basic research, stating that applied research is only possible when you have a solid foundation of basic research.

 

Day 2

Dr. Hirofumi Seike, University Research Administrator, Management Associate Professor, Tohoku University presented ‘University Internationalization and its Impact’.

Dr. Seike raised internationalization as a challenge, and why? Tohoku commits to providing students the best quality international perspective possible, and the university believes international experience will ensure a high quality education and research,  as well as expanding their human networks.Many global issues can only be solved through international collaboration, but Japan encounters a problem of students not wanting to study abroad. In this sense, he feels Japan is falling behind.

In terms of research, he feels that Japan has stagnated, while other Asian countries are increasing their presence. The government shares a strong sense of urgency which leads them to initiate multiple globalization projects and set targets such as to include 10 universities in the top 100 in world rankings.

Dr. Seike introduced one of the government initiatives, WPI,which aims to establish world-class research institutes. Tohoku University was chosen as one of them. WPI empowered the awardees to have their own governance,which allowed competitive recruitment to assemble world-class innovative scientists that can lead from basic research to industry application.

 

"WPI established a special zone within the existing university framework.  It’s a new approach… not just the expansion of the existing system. It should be the showcase of the best research… the best of the best.”

Dr. Hirofumi Seike, Tohoku University

 

 

 

 

Professor Paul K.H. Tam, Pro-Vice-Chancellor and Vice-President (Research), University of Hong Kong (HKU) presented ‘Research Excellence and Internationalization at the University of Hong Kong – Striking the Right Mix of Metrics and Faculty Expertise’.

Professor Tam described the University of Hong Kong (HKU) as an institute of great heritage but with many structural issues that needed to be resolved – and shared the ways they overcame these challenges. It was a transformation from a predominately teaching university to a comprehensive research university.

A major motivation for institutions of Higher Education is competition, and the introduction of other universities in Hong Kong ‘awoke the giant from a deep sleep’. While the transformation was also self-motivated, there were important external factors which came from the government; the establishment of The Research Grants Council followed by the introduction of the Research Assessment Exercise.The previous funding system allocated 75% of the money into a recurrent grant that supported continuity and sustainability. Distribution was based on student places (75%) and only 25% was related to research. The government changed the system to drive major change, and allocation is now judged using performance indicators.

What does it mean to be a ‘world-class’ university? HKU agreed upon having a tradition of research excellence with internationally competitive staff and more importantly, a strong culture that will attract students globally as the choice of institution for those who want a career in research. HKU has been very successful despite there being 8 institutions. They are responsible for over half of large program grants, and have the top position in every assessment indicator, be it grant amount or research output.

Talking about university transformation, Professor Tam spoke about the guiding principles of providing an enabling environment for researchers and respecting academic freedom by keeping a bottom up approach which is top facilitated.

“What I consider the greatest asset of the university is human resources, the talents. It is the role of the university leaders to provide an enabling environment for the researchers – this is my guiding principle. The other principle I have is that we have enjoyed the principle of academic freedom and we respect that and continue to cherish it. To respect that means the approach is bottom up. There can be a lot of debate between top down and bottom up approaches. We have kept a bottom up approach but introduced a top facilitated bottom up approach.”

Professor Paul Tam, HKU

 

Dr. John Green, Life Fellow, Queens’ College, University of Cambridge, presented ‘Evidence Based decision making in academic research’.

First Dr. Green created context by talking about increasing interdisciplinarity and internationalization in science, and the increasing demand for evidence to evaluate outcomes and justify funding expenditure. He spoke about how Imperial College evaluates interdisciplinary institutes that work cross-departmentally every 3 or 4 years, and on what basis they close institutes down.

Dr. Green touched on the potential of getting lost in the avalanche of data available today and the importance of getting meaningful information from the data. It is important to understand where the strength of an institution lies, where to focus its strategy, who to collaborate with. He explained the importance of due diligence about specific partnerships, the need to find ways to connect researchers and facilitate the mobility that will create the collaboration. He stated firmly that these things do not happen bottom up, that there is a need to facilitate them based on evidence to inform the facilitation. At Imperial, he created and used a systemthat presents the research performance dashboards at the departmental level.

 

“The world has changed now, and if only some of the systems which are available to you now were available to me then, I would not have re-invented the wheel… Pure has now come into the market, which does exactly what we were trying to do, but it does it better. It is a system which sits on top of your internal IT systems and harvests information from it and provides you with dashboards, and that is exactly the concept that I have been talking about.” (Figure.4)

Dr. John Green, Life Fellow, Queens’ College, University of Cambridge

 

 

Figure 4 – Example of a dashboard in Pure which shows research outputs, journals and activities for a university

Having spoken about the metrics, he pointed out the need to standardize the definitions and methodology involved in generating metrics, because everyone tends to do it differently which means that the results cannot accurately be compared. How can we compare the number of researchersif each university defined researcher counts differently? The Snowball project, a non-commercial initiative in which Dr. Green and Elsevier are involved, resulted in agreed upon methodology for these metrics that are endorsed by a group of distinguished UK universities.

“I don’t want to give the impression that metrics are everything. Metrics are one of a number of ways to come to a judgment… and help you come to a view of something. In no sense are they a way to navigate your car. A Satellite Navigation system is something that tells you what the best route is and how you might change the route if there are traffic jams… But you have to decide which is the best route for you, based on that information and other information too (for example where you want to go for lunch, do you want the prettiest route or the autoroute). That is why we need other measures such as peer review to complement metrics.”

Dr. John Green, Life Fellow, Queens’ College, University of Cambridge

 

Dr. William Gunn, Head of Academic Outreach, Mendeley, presented ‘Innovation – Scientific and technical foundation development for altmetrics in the US’.

Dr. Gunn spoke about totally new metrics, which may compliment, or arguably even be an alternative to traditional metrics, hence the term ‘altmetrics’. He suggests that new forms of scholarship need new metrics. Altmetrics are faster to accrue compared to citation data, and they use research and social media data that is totally outside of the traditional research metrics. Altmetrics for impact include usage of articles, peer-review such as via post publication commentary services,and social media activities such as discussions on blog posts to measure attention and impact work had given to others. He drives home the point that there are many ways to look at the overall influence of a paper or group of papers, and that citations are just a tiny fraction of that.

“There are 125 times more downloads of papers (than citations) and a universe of social activities, that are being aggregated…, so there is a lot more data out there that we can gather, work with and use to understand the impact our work is having”.

Dr. William Gunn, Head of Academic Outreach, Mendeley

 

 

 

 

Nonetheless, altmetrics are not without challenges with regard to transparency and consistency. There are different services that provide altmetrics such as Plum Analytics, Impact Story, and Altmetric, and if you query them all on one specific DOI, there are differences in the metric values reported back, which leads us to question which value is correct. There are also problems with identity attribution if researchers use a fake identity, and finally altmetrics can also be gamed, although it is difficult if people make use of many sources and many different metrics.

Looking back, the conference was fascinating in that the speakers and participants alike were passionate and often candid in sharing their views and experiences, resulting in lively discussions, which we all could learn from.

References

(1) ELSEVIER (2013) “International Comparative Performance of the UK Research Base – 2013”. Available at: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/263729/bis-13-1297-international-comparative-performance-of-the-UK-research-base-2013.pdf

 (2) Franceschet, M., Costantini, A. (2010) “The effect of scholar collaboration on impact and quality of academic papers”, Journal of Informetrics, Vol. 4, No. 4,  pp. 540-553.

 (3) Prinz, F., Schlange, T., Asadullah, K. (2011) “Believe it or not: How much can we rely on published data on potential drug targets?”, Nature Reviews Drug Discovery, Vol. 10, No. 9, pp. 712-713.