Dear friends of Tomorrow’s Teaching,
Welcome back to another post on how AI can help us save time and effort in preparing for our classes. In the past few weeks, we talked about how to brainstorm your course with AI:
how to get AI to inspire your students:
and how to make it help you in preparing your exams and creating model answers and grading rubrics:
These posts all used some version of ChatGPT and the examples will work in more or less the same way with any one of the common Large Language Models.
But today, I want to look at a different AI tool — one which I find amazing, and that allows us to do things that ChatGPT cannot do in the same way. It is a tool that has received very little public attention, compared with ChatGPT, but I find that it can be very useful both for researchers and instructors and for students who can use it to cheat in ways that were not easily achievable before.
I am talking about Google’s NotebookLM, a program that allows you to question a chat interface on the content of a number of sources that you provide. This can be a great help when you are preparing topics that require a lot of reading.
This is why today’s post will be the first in a series of three. First, I will show you how NotebookLM works and what you can do with it. Then, in the second post in this series, we will look at how students can use NotebookLM to cheat. And finally, we will see what you can do, as an instructor or teacher, to stop students from cheating on their assignments using AI tools.
So let’s jump in!
Google’s NotebookLM
NotebookLM was introduced as an experiment by Google in July 2023, and it’s still in an experimental stage, as a label on the top of the page informs us. This means that you shouldn’t rely on it too much — Google is well-known for killing off experiments that don’t perform as it would like. So use it while it’s there, but don’t save big amounts of research data in it in the hope that you will be able to access your sources in ten years’ time. I keep my files safely on my own drive, and only upload what I want to ask NotebookLM about at a particular time.
Here is what the program looks like when you start it:
You have a series of example notebooks that you can explore at the bottom of the page, and above it are all your own notebooks. You would usually create one notebook for every topic that you are researching. A notebook is, essentially, a collection of sources. So if you are writing a number of articles or papers on the same set of sources, then I’d put those into one notebook. If, instead, you are writing five articles on five different topics, utilising totally different sources, then you may want to create five notebooks. A notebook, as I understand it, is best seen as a library shelf for a collection of related sources.
So let’s say that we are trying to get an overview of Kant’s Critique of Pure Reason and, generally, his epistemology. Let’s create a new notebook for that. And this is what our new notebook looks like:
What I initially found confusing is the distinction between “sources” and “notes.” As I understand it now, “sources” are the books and papers you want to query your NotebookLM about. “Notes” are the results of these queries and any other insights that you have while you work, and that you want to save and remember for later: the results of your research.
Adding sources
Let’s add a few sources now. Click on “Add Source” and you can choose from where to upload your materials. I had the best success with PDF or text files. Weirdly, web addresses don’t seem to work as sources. So if you want to add a Wikipedia article as a source, you’ll have to “print” it as a PDF file and then upload that PDF file to your notebook.
Let’s do this. I’ll upload a number of articles from the web:
The Wikipedia articles “Immanuel Kant” and “Critique of Pure Reason.”
The Stanford Encyclopedia of Philosophy articles “Kant’s Account of Reason” and “Immanuel Kant.”
And the Internet Encyclopedia of Philosophy article “Immanuel Kant: Metaphysics.”
Unfortunately (stupidly, one might be tempted to say) one cannot upload more than one source at a time. So we have to do the upload dance five times, once for each source. Such details show that NotebookLM is an “experiment” rather than a thought-out product. There is no good reason why one shouldn’t be able to upload all of one’s PDF files at once, or why the address to a Wikipedia article couldn’t be accepted instead of a PDF file of the same article.
After the uploads are finished, we’ll have all our sources on the left sidebar.
Source guides
If you click on any of the PDF files, you will get a “source guide” window:
This gives you a summary and a list of topics on the right. Click on any topic button, say, “Transcendental Idealism,” and a summary of what this source says about the topic will be displayed in the notes area:
At the bottom of the note is a button saying “X citations.” Clicking on that button will expand a list of citations, and clicking on any of these numbers will show the source on the left side where the respective information comes from.
This is great, because it immediately solves one of the greatest issues with LLMs: hallucinations and the unreliability of their results. In the case of NotebookLM, you can immediately check where the information came from and see the source at just that point.
The query interface
Let’s now use the “chat” interface. NotebookLM will already suggest queries, but we can also ask whatever we feel like. If you look carefully, at the left of the chat entry, you will see the number of sources that will be used to answer your query.
You can switch particular sources on and off, by using the check box that is attached to each source on the left side. Now make sure that we have them all activated. The number should read “5 sources.”
If you read the summary NotebookLM gave us, you will see that it contains this sentence about transcendental idealism:
It has been criticized by both empiricists and rationalists.
Let’s now ask it why this is the case:
Why exactly has transcendental idealism been criticized by both empiricists and rationalists?
Again, one can check the sources for verification by clicking on the “citations” button under the note. If you want to save a note, you can pin it using the pin on its top right, which will make it into a note.
You can also add a note of your own by clicking on the “add note” button on the top right of the notes area. This will open a little editor, where one can freely type anything.
Hallucinations?
Now let’s try to get NotebookLM to give us wrong information. We’ll ask about something that is not covered in the sources.
What was Kant’s relationship to his first son?
And we get the glorious answer:
I apologize, but the sources you provided do not contain any information about Immanuel Kant's relationship with his first son.
This is spectacular because it is precisely what LLMs are usually bad at: knowing the limits of their own knowledge. NotebookLM will strictly only use the provided sources, so if something is not in the sources, it won’t appear in the output. This makes it possible to even research only a particular viewpoint on a topic, by selecting the sources one provides to the notebook. If I, for example, use only, say, climate change denialist sources, I will get a climate change denialist reply. This is excellent for researching particular positions on their own terms.
Similarly, if I ask:
Did Kant have any children?
I get the reply:
The sources provided do not mention Kant having any children.
Comparing sources
What topics does the Stanford Encyclopedia of Philosophy mention about Kant that are not included in the Wikipedia article about Kant?
I got a long reply, which I must admit that I did not check for correctness. Try it out yourself. But if the answer was correct, it would be great to be able to compare two sources and see which topics are discussed in which one and not in the other.
Another thing we can do is hunt for particular information across sources:
Which of the sources has a table listing all the Categories?
Kant Metaphysics _ Internet Encyclopedia of Philosophy.pdf has a table listing all the Categories.
This is correct. Let’s try another:
Which source gives the easiest to understand explanation of the difference between Transcendental Aesthetics and Transcendental Analytic?
It not only names the source, but extracts and presents the answer at the same time, although I’m not sure that this answer is the clearest possible. At least it’s not immediately understandable to the non-specialist. I believe that ChatGPT would make a better job of explaining the concepts, but then, it likely wouldn’t be able to compare the quality of the two sources.
Grading assistant?
As a final experiment, I tried to feed two made-up student papers into NotebookLM. If it can judge “easiest to understand” across sources, perhaps it can also judge the quality of student work?
Both papers were short, about two pages each. Paper 1 was written to be a very good paper, well-structured, written in good English, about David Hume’s Skepticism. The second paper was on a different topic: a hastily written, under-researched, mostly common-sense paper on Locke’s social contract theory.
My prompt was:
Comparing the two essays in the sources notebooklm1.pdf and notebooklm2.pdf, which one is better written and should get a higher grade and why?
I got this reply:
The essay in notebooklm1.pdf is better written and should get a higher grade than the essay in notebooklm2.pdf because it is more well-organized, provides a more thorough analysis of the topic, and uses more scholarly sources.
Organization: The essay in notebooklm1.pdf is organized into clear sections, each of which addresses a different aspect of the topic. The essay in notebooklm2.pdf, on the other hand, is more difficult to follow and lacks a clear structure.
Analysis: The essay in notebooklm1.pdf provides a more thorough analysis of the topic. The author does a good job of explaining the different arguments for and against the state of nature and provides a nuanced understanding of the concept. The essay in notebooklm2.pdf, on the other hand, is more superficial and does not provide as much insight into the topic.
Sources: The essay in notebooklm1.pdf uses more scholarly sources to support its arguments. The author cites works by John Locke, Immanuel Kant, and other philosophers. The essay in notebooklm2.pdf, on the other hand, does not use any sources to support its arguments.
Overall, the essay in notebooklm1.pdf is a more well-written and well-researched essay than the essay in notebooklm2.pdf. It provides a more thorough analysis of the topic and uses more scholarly sources to support its arguments.
You see what it did here? “The essay in notebooklm1.pdf provides a more thorough analysis of the topic. The author does a good job of explaining the different arguments for and against the state of nature...” But file 1 was not about the state of nature. That was file 2. File 1 was about David Hume’s skepticism!
The rest of the evaluation is correct, but a mistake like this, confusing the topics of the two papers, makes the result worthless and introduces a new worry about the quality of NotebookLM’s cross-source comparisons. Is it really true that the SEP gives better explanations than the other sources? Or did it confuse its sources again?
Remember: It’s not magic. It’s an LLM, after all. You always have to check their output!
Summary
I think that NotebookLM, as long as it exists, can be an excellent tool to manage one’s sources for particular projects — if one checks the output for correctness! But this is the case with all AI at the moment.
Whether it is research for a class one is teaching, or reading a long book and trying to find particular passages, summarising knowledge from multiple sources, or hunting for insights across papers — NotebookLM is the assistant every reader of complex source material will find a use for.
The tool is free to use with a Google Account. You can access it here:
Some users on the Internet commented that you don’t need NotebookLM if you train your own version of ChatGPT with your own documents.
I don’t agree.
For one, NotebookLM is available everywhere in the world (via VPN, if outside the US) for free, while ChatGPT can only be trained with a premium account, which is not available in many countries (for example, in mine).
Second, I don’t know how a custom-trained ChatGPT would respond if asked something that is not in its sources — will it say that it doesn’t know, or make something up? I am particularly happy about the clarity with which NotebookLM states that it cannot find that information in its sources. It still has other places where it will hallucinate and go wrong, but at least one source of errors is eliminated by design.
Finally, the combination of literature research with the integrated note-taking interface is done well enough. One can always imagine improvements, but as it is, it is already quite useful. One can save replies, add one’s own, and later use these notes in writing one’s paper, lecture or article.
I hope this was useful and that you will give NotebookLM a try. Tell me in the comments what you think!
Have a nice weekend,
Andy