Training ISS premaster Academic Skills: Evaluating sources

1. How to determine the relevance of sources?
2. How to determine the scholarly nature of sources?
3. Crapmap
4. Impact factor and Journal Citation Reports (video)
5. Peer review (incl video)

Once you have found a set of sources it is important to evaluate whether they are suitable. You assess whether they suit your subject sufficiently (content related, relevance) and you assess whether the quality is sufficient (scientifically suitable). 

Determine for your found sources whether they are relevant to your topic and whether they are of a scientific nature (be aware that not all your sources need to be scientific at all times, for example sometimes it is very useful to use reports or government documents).

The assignments below can help you on your way:

A. Find out from your found book where the authors are working and / or what else they have written.

B. Find out from your first-mentioned article whether the journal the articles is published in is peer reviewed. This should be on the site of the journal itself, but also, for example, in the Utrecht University Library list of e-journals.

C. Find our from your last-mentioned article:

• the quality of the references;
• whether the research method followed and the method of data collection are explicitly stated.

D. Check the website of the journal Child Development. Here you can find information about the impact of the journal (JIF and ranking). Also check the homepage of the journal Clinical Psychology Review .Here you will find, next to the JIF, also other forms of journal metrics and also information about impact in social media (also called altmetrics) of articles. For example take a look at the article 'What is compassion and how can we measure it? A review of definitions and measures' and see what the social media impact was on Twitter and Facebook. 

To determine if a source is relevant, you may try to answer the following questions:

1. Does the sources offer you information that is useful to answer your main questions and sub-questions?
2. To what extent does the main question of the source you found match with your own questions?
3. Does the source answer your whole question/sub-question or only one aspect?, for example only a period, or a person or a group?
4. Does the information provide a general overview? Or does it deal with exceptions or specific areas?;
5. Is the (social / cultural / historical) context of the research object the same as in your case?
6. When was the piece published and when was the research written about executed?

Think that you will rarely find a source that provides a complete answer to your main questions and sub-questions and that gives a report of the exact same research or problem you are working on. 

Want to read the full text: the UBU-link will give you access to publications from the online collection of the Utrecht University Library.

Visit the LibGuide Evaluting sources  for more information.

The crux of science lies in the extent to which an author/researcher performs his work objectively and makes it verifiable. In determining the quality and scientific nature of sources you may start from three kinds of checks:

1. Check by others, preceding publication
   • editors: editors of scientific journals are stricter than editors of non-scientific journals
   • publishers: some publishers only publish scientific books
   • peer review: most scholarly journals but also some book publishers ask experts for a (single or double blind) judgement before publication
   • search engine/online bibliography: some search engines only include articles from high-quality, peer reviewed journals (for instance Scopus and Web of Science
   • financers: some journals demand to know who funded the research
2. Check by others, after publication
   • reviews (in the case of books): are the reviews positive?
   • citations (mainly in the case of articles): is the piece cited often (taking into account how long it has been available) and more importantly:what is being said?
   • open annotation and post publication peer review in systems like PubPeer and (for preprints) Peer Community In and comments at the preprint platforms themselves.
3. Check by yourself
   • are author and date of the text mentioned (particularly in the case of webpages)?
   • affiliation of the author: the author's job may tell you more about the scientific level, for instance if the author is employed by a good university
   • is the target group mentioned: (in particular in the case of websites and reports)
   • presence of explicit research questions and conclusions
   • presence of an explanation of the method used: how was the research conducted, where are the data coming from?
   • presence of enough and high-quality literature references or notes: what insights are used?
   • language level and well organised text

A useful tool to evaluate the quality of your sources is the CRA(A)P test, developed by the library of California State University, Chico. The CRA(A)P test consists of a list of of questions that help you to determine if the (online) information that you found is useful and trustworthy. Depending on your specific situation items can be more or less relevant.

This checklist is used as the basis for the CRAPMAP:

Currency
The timeliness of the information

 When was the information published or posted?
 Has the information been revised or updated?
 Does your topic require current information, or will older sources work as well?

Relevance
The importance of the information for your needs

 Does the information relate to your topic or answer your questions?
 Who is the intended audience?
 Is the information at an appropriate level (not too simple or too advanced)
 Have you looked at a variety of sources before determining this is one you will use?
 Would you be comfortable citing this source in your research paper?

Accuracy
The reliability, truthfulness and correctness of the content

 Where does the information come from?
 Is the information supported by evidence?
 Has the information been reviewed or refereed?
 Can you verify any of the information in another source or from personal knowledge?
 Does the language or tone seem unbiased and free of emotion?
 Are there spelling, grammar or typographical errors? [Second task or achievement]

Publication
The source of the information

 Who is the author/publisher/source/journal/sponsor?
 What are the author's credentials or organizational affiliations?
 Is the author qualified to write on the topic?
 Is the publisher/journal scientific? Are they reputable?
 Is there contact information, such as a publisher or e-mail address?

Metrics*
The impact a source has on the scientific field

 Are you able to find metrics for the source?
 Is the source cited by current research?

Altmetrics*
The impact a source has in general

 Are you able to find altmetrics for the source?
 Is your research impacted by how well the source is received?
 What does the public reception of a source indicate?

 * For information on metrics and altmetrics see our LibGuide Research Impact & visibility: traditional and altmetrics

Purpose
The reason the information exists.

 What is the purpose of the information? Is it to inform, teach, sell, entertain or persuade?
 Do the authors/sponsors make their intentions or purpose clear?
 Is the information fact, opinion or propaganda?
 Does the point of view appear objective and impartial?
 Are there political, ideological, cultural, religious, institutional, financial or personal bias?

Read more about plagiary, citing and paraphrasing on the next page.

Peer review is the evaluation of work by one or more people with similar competences as the producers of the work (peers). It functions as a form of self-regulation by qualified members of a profession within the relevant field. Peer review methods are used to maintain quality standards, improve performance, and provide credibility. In academia, scholarly peer review is often used to determine an academic paper's suitability for publication.

source: Wikipedia 

See also this video below:

Library NCSU. This video is published under a Creative Commons 3.0 BY-NC-SA US license

Journal metrics are used to evaluate impact and quality of journals. There are several journal metrics, the journal impact factor (JIF)  is one of them.

The video provides information about the way the impact factor is calculated and how to find impact factors in the Journal Citation Reports. An Impact Factor is a figure that does not say much in itself. You must always view journals and their impact factors in the context of comparable journals (rank).