Resource hub for PHSA researchers interested in using artificial intelligence (AI) for research tasks and for conducting research into the use of AI.
The content for this page in supported by the AI in Research Working Group, who provides guidance and support for PHSA researchers interested in using AI to assist in completing research tasks, as well as those interested in conducting research into the use of AI. The Research Hub will be updated regularly as new information and guidance is available.
The AI in Research Working Group developed
guidance to support safe and responsible use of generative AI (genAI) as a tool to enable research work and complete research tasks. There are many potential uses of genAI in medical and health research and these uses may involve both significant risks and benefits. PHSA seeks to promote and encourage the use of innovative tools to support research, and to encourage innovative research, including in the field of AI.
The following use cases further describe potential benefits and risks of using genAI as a tool to complete research tasks. In each use case, there are considerations of benefits, risks and mitigation strategies based on current knowledge.
Jill is a graduate student and working with her supervisor on a Canada Institutes of Health Research application. She is asked to draft a few paragraphs for an introduction on the topic of long-term effects of the COVID-19 pandemic restrictions on school performance. Jill decides to use ChatGPT to help her get started and get ideas about what to write.
Benefits
Risks
-
Uncertainty / misinformation: the output may be very general and lack citations. Some genAI tools allow you to upload a set of papers and ask questions specific to those sources. Output must still be carefully reviewed as hallucinations (made up facts) are always possible with these tools.
-
Mitigation: all listed facts or statements in the output must be verified and references found. Prompting ChatGPT to provide sources will result in a list of potential websites to review but these will not be linked to any specific statement in the output. Literature searches are more efficiently performed in a reputable database.
-
Privacy: any data input into a genAI tool may be stored by the parent company for further model development or monitoring. It is often unclear what is stored and who may get access to it.
-
Mitigation: never put sensitive information into a genAI tool with unknown privacy agreement. Find an alternative strategy to achieve your goals or consider using a local/institutional genAI or a subscription that meets PHSA privacy requirements.
-
Bias: most genAI tools are developed on corpuses of information scrapped from the internet and these sources are not fully disclosed. It is often unclear what the resultant bias in the output may be.
-
Mitigation: all outputs must be carefully considered for possible racial, gender and other biases. It is up to the end-user to take responsibility for output.
-
Plagiarism / disclosure: many genAI tools do not provide clear citations or source information and will directly copy statements from books or articles in the output. Without verifying output and identifying reputable sources you run the risk of plagiarizing other people's work.
-
Mitigation: all listed facts and statements in the output must be verified and appropriate references found.
Philip is an undergraduate student working on a research project that aims to create an image classifier that can classify whether the chest x-ray shows a pneumothorax. He is familiar with training image classifiers in TensorFlow but wants to implement in PyTorch. He decides to use ChatGPT to help set up the training function.
Benefits
- Efficiency: saves time by automatically writing boilerplate codes
- Provides different approaches to a problem
- Can help user learn about syntaxes of a new programming language or libraries
Risks
-
Incorrect code: if the prompt provided is not clear in terms of what the user wants the code to do, the large language model (LLM) can produce a code that provides a different logic than desired. Like hallucination in text-generation context, the LLM may produce results that incorporate non-existent libraries or tools, or incorrect logic, despite giving an appropriate prompt. In addition, the logical errors may be subtle and difficult for the user to debug.
-
Mitigation: always manually check the logic of the code generated by the LLM.
-
Outdated code: ChatGPT's internal knowledge base may not be the most up to date, so it may not be able to work with the newer libraries or tools. Moreover, it may propagate reliance on outdated practices or code.
-
Mitigation: being aware of how frequently the LLM is updated provides a rough estimate of how up to date the code it generates could be. User may need to refer to other documentations for the newer libraries and tools.
-
Efficiency: LLM may produce a code that is logically correct, but computationally inefficient.
-
Mitigation: manually review code to optimize any inefficiencies.
-
Security: the LLM may suggest code that is functional but contains security vulnerabilities.
-
Mitigation: there are LLMs which are specifically trained to detect and fix security vulnerabilities. However, it is important to manually review and test for security vulnerabilities and consult a security expert, if required.
-
Privacy: any data input into a genAI tool may be stored by the parent company for further model development or monitoring. It is often unclear what is stored and who may get access to it.
-
Mitigation: never input sensitive information into a genAI tool with an unknown privacy agreement. Find an alternative strategy to achieve your goals or consider using a local or institutional genAI or a subscription that meets PHSA privacy requirements.
Giuseppina is putting together a grant due in a few hours. She is done with most components, but still has to write a lay summary that can be understood by the lay public. She takes the abstract for her grant and supplies it to ChatGPT with the following prompt: "Please rewrite the following scientific abstract so that it can be understood by the lay public: [inserted abstract]"
Benefits
- Quickly produce a lay abstract
- ChatGPT supplies some ideas on how to word certain concepts in a widely accessible way
Risks
-
Inaccuracy: the output may have factual errors introduced, given that it is trained on a general language corpus and not specifically on scientific content. It may summarize concepts incorrectly or reword statements that are no longer correct.
-
Mitigation: all output must be carefully checked for scientific accuracy.
-
Language use: the output may be rewritten to be too simple, or may still be too specific and not appropriate for a lay audience.
-
Mitigation: the output must be checked and compared with the tone, complexity and language use that the author would normally use. ChatGPT's judgement should not be substituted for the user's.
-
Plagiarism / disclosure: genAI may directly plagiarize others when generating its content. Even if the subject matter is specific to your grant, generated sentences could still be inadvertently plagiarized.
-
Mitigation: if ChatGPT has significantly reworded sentences or paragraphs, consider rewriting them in your own style, using some of the suggested wording if desired. If in doubt, online tools for plagiarism detection can be used, though these are not foolproof.
Jennifer is working on a proposal for an infrastructure grant that involves building a new research consortium. She wants to create a logo for the consortium to develop some branding and create a better sense of coherence. She finds an AI-based logo generator online.
Benefits
- New logos are created quickly and easily
Risks
-
Plagiarism: genAI-based image generators are trained on existent images. A new logo may be very similar to that existent images that were used for training.
-
Mitigation: use an image-searcher to compare your image against others (e.g., https://images.google.ca). As possible, try to avoid very simple logos that might be similar to what already exists.
-
Terms of use: image generation tools may allow images to be used with restriction and may retain rights to use the image in certain contexts.
-
Mitigation: carefully examine Terms of Use for the product, noting if there are any restrictions, or if users retain full rights to the images. If in doubt, use a different product, or consult legal resources.
-
Inappropriate content: such tools are trained on images across the internet and can include a wide variety of content. Generated images may contain components that have special meanings to groups of people, even if not obvious to yourself.
-
Mitigation: ensure that a wide variety of others review generated images before they are publicly shared, especially if the logo contains elements that seem distinct. Have a low threshold to replace elements that may possibly be misunderstood.
-
Bias: if your logo contains representations of humans, genAI models may portray them by furthering stereotyping with respect to gender, race, class, profession or other attributes. However, models may also try to correct for this but lead to portrayals that are incorrect.
-
Mitigation: consider whether your images further both positive and negative stereotypes. Consider revising your prompt to add diversity and combat such stereotypes. Ensure that portrayals are still consistent with intention.
Jacob is reviewing several applications for a research institute graduate student travel award. He wants to use ChatGPT to select the strongest application. The rubric provided for the grant review asks reviewers to give each application a score out of 5 points in the following areas: candidates track record of academic performance and productivity; relevance of the travel to the trainee's research; and quality of proposed research or training activity associated with the travel.
He copies and pastes each application with the applicant's CV into ChatGPT before providing the following prompt: "I am going to provide you with a number of grant applications. Summarize each application using the following criteria: number of candidate's publications; total impact factor of publications; clarity of writing; inclusion of a research or training plan in the application; and inclusion of a statement linking proposed travel to the applicant's research goals."
Benefits
- Quickly pulls out some relevant information for review
- Summary statements could be used to report back on scores
Risks
-
Bias: if you include names, gender or age of applicants, models can introduce bias in how these factors are assessed due to the underlying bias in development data. This could lead to reduced scores or lower assessment f of quality for applicants that are unfair.
-
Mitigation: do not ask for subjective review of quality of proposals using a genAI tool. Only very clearly defined quantitative assessment categories could be used but output still needs to be double checked for accuracy.
-
Inaccuracies: there is a known issue for genAI tools to hallucinate or make up facts. The output may therefore have factual errors introduced, given that it is trained on a general language corpus and not specifically on scientific content. It may summarize concepts incorrectly or reword statements that are no longer correct.
-
Mitigation: all outputs must be carefully checked for accuracy.
-
Lack of reducibility / transparency: the models used for genAI tools are continuously being revised making reproducibility of results often impossible even with the exact same prompt. This ongoing change in models also means we cannot fully know how the prediction or output is being generated.
-
Mitigation: use of a genAI tool to support grant review should always be disclosed to the funding body. Justification for funding decisions must be clear and many funding bodies may not support the use of genAI in this process.
there is a know
- Ask genAI tool to provide list of references, including hyperlinks, to be able to quickly assess accuracy and quality of source information (possible in some tools)
- When editing, ask for bullet point suggestions for edits rather than a direct re-write of provided text. This allows you to maintain control of all changes and verify the quality of the final written product.
- Describe your data in the prompt and ask for example code that would work with the type of variables your dataset includes.
- Use specific language to describe the audience for the intended output.
- Use an iterative process to keep refining the results until the desired result is achieved.
- For some image generators you can specify the licensing type you are interested in. For use in public or scholarly communication this should typically be for public use/domain, or "Creative Commons" known as a Creative Commons Zero (CC0) license type. See
here for an example of this.
- Check with the funding body first if they will accept review supported by genAI.
- Only include clear and quantitative categories of review so that output can be verified and checked for accuracy.
