Well, thank you. I'm uh absolutely thrilled to be here. I'm Andy Hall. Uh I'm a professor here. I'm uh this is the second time I've given one of these talks. The first time my mom was in the audience. Uh she's not today, so the pressure is totally off. So, I'm actually uh on leave right now working at Anthropic, but I haven't gone very far. and I'm delighted to be here with you all today to talk to you about uh what's going on with AI and politics
and things like that as well as how it's entered our teaching here at the GSB. There couldn't be a better place uh to be doing this than the GSB obviously. So, I'm going to start keeping my eye on the clock because I want to leave a lot of time for questions. I'm going to tell you just like a little story of kind of what got me into AI and and it'll give you a sense of where my head is at. I'll tell you just a little
bit about the research that we're up to here, a little bit about the teaching uh and then wrap up. So I'm a political scientist by training and most of my research historically involved collecting largecale quantitative data so from the web or from election officials or whatever and analyzing it. And uh in 2020, my lab wrote a paper during the pandemic about the question of uh how universal vote by mail affected turnout and Democrat versus Republican election outcomes. You might remember this was a very contentious topic
during the 2020 election. It's in fact still an issue now. And during the pandemic, we wanted to do sort of urgent policy relevant research. So by the standards of that time and by the standards of academia, we did a very rapid project which took us only three months in which we collected a bunch of county level data on which counties in mostly western states had rolled out this policy and how it had affected their elections. Now, fast forward to last holiday break, and we had been
hearing for a while that AI was getting better and better at everything, including helping us do our research. And I uh was very excited about this. I've been trying for a while to find ways to work with AI. And something very exciting happened. Maybe some of you experienced this yourselves, but over last holiday break, so we're we're coming up on a year now. Uh a new model came out from Anthropic. I was not working there at the time. Uh that was much much better at doing
this kind of automated code stuff through what's called Claude Code. And people started going crazy and and sharing all these examples of things they had gotten Claude Code to do completely autonomously. I rushed to social media to announce how excited I was about this. I posted to X, which is where everyone talks about AI, and I said, "Oh my god, this is going to completely change how we do political science research. You can use cloud code now to do all this research autonomously for you." This
was X. So obviously, what happened immediately was people started shouting at me and they said, "Uh, you're an idiot. You know, there's no way Cloud Code can do that." So I said, "Okay, I'm going to prove it to you." And I took this paper of mine from 2020. I gave it to Claude and I said, "Claude, here's this old paper. Here's the code and the data. Would you please," I'm always very polite to Claude, I said, "Would you please uh take this old paper and update
it?" You know, it's kind of crazy that we write these papers, the world goes on, more information is gathered and collected, and yet we don't go back and update these papers. And that's true not only in academia, but in corporate data science as well. And so I said, "Claude, will you do this for me?" And then I sat there and I watched uh for 45 minutes is all it took. Claude on my computer. So if you use Claude code, it's not like chatting with Claude on
your phone. It actually kind of like controls your computer and it can do stuff. So I sat there and I watched as Claude went to all the government websites and downloaded for me the new election data since 2020. put it together, ran wrote the code on my computer to run the analysis, made the tables and figures, and even wrote a little paper. Uh, and so this was astonishing. I posted that to X. It got a ton of attention, a lot of media coverage that this is
like a new way to do research. And then obviously I got screamed at again, this time because of course people said, well, it's just all AI slop, right? It's just all hallucinated. Uh, so I hired a graduate student. His name is Graham. Graham is an election administration expert at UCLA. And I said,"Graham, would you please I'm nice to Graham, too. I said,"Graham, would you please, would you please update this without using AI and then we'll compare them to one another." So, he did that. Took him
a couple days and then we compared them and we found, you know, Claude did pretty well. There were some mistakes uh in Q&A. can tell you about some of them if you're interested. But overall, Claude did really well. And that I think raised some very profound questions for me because it was clear on the one hand I did need to oversee Claude. It wasn't that Claude could just go and do this. Uh but on the other hand, the timesaving of having Claude do it and then
check it was massive compared to having done it without Claude. And so I had I took two things away from this. The first was, you know, if this is going to uh do this for all of my research, like there's no time to waste. I need to starting today, I'm never going to go back to the old way of doing my research. I'm going to do everything with AI. So, I've built a lab. I'll tell you a little bit about it where all of our work
is organized around working with AI to do our research. The sec and and my teaching as well, which I'll tell you about. The second thing I took away from this was, no offense to political scientists like me, but if this is going to change political science this profoundly, what is it going to do to the rest of the world that might matter a little bit more than us? Uh, and so I decided, you know, pretty immediately, I'm not just going to use these research tools to
do the same kind of research I was doing. I'm gonna use them to study AI because if we want to influence the path of society and politics and our democracy, it's going to be hugely wrapped up in what happens with AI. And so those are the two decisions I made. I've been on sort of a strange and interesting journey since then. That was in January of this year. Uh as I said, now that journey has taken me uh to Anthropic where I'm working on leave right
now. Obviously, I'm here speaking in my personal capacity. Um, and I want to tell you a little bit about what we're doing, what I've been thinking about, and the lab, which I'm still running while I'm on leave. Um, as I thought about AI and what it might do to politics, one of the things that occurred to me is that very understandably, we're mostly talking these days about the risks of AI with respect to politics, and we're not talking very much about the opportunity. And I wanted
to do something to think of to to offer the positive vision but then also realistically assess the problems that we're facing along the way. And the more I thought about it, the more I decided that I think the right analogy is the printing press. So the printing press was a technology that as Condors argued was massively essential critical to the rise of modern democracy because it made it cheap and easy to broadly distribute information. This was Condors's argument and that really profoundly changed the world. It
made all of society in some sense much smarter and that laid the foundation for modern democracy, modern representative democracy. It did not happen without, you know, substantial disruption that we would like to avoid this time. So, we're talking several hundred years of religious conflict, war, uh, genocide, anything bad you can think of, right, happened. In fact, Condors wrote his argument about how important the printing press was from prison, uh, where he was because of, you know, the aftermath of the French Revolution. I think AI does
something similar but much more profound than the printing press because like the printing press made it easy to copy and distribute information. AI makes it cheap and easy to distribute intelligence. And so if you look at what does it take to to run an enlightened society? Well, you don't just want everyone to have information. You actually want them to be able to process that information and turn it into decisions. And if we were to get AI right, then it should give for each of us a
sort of infinitely patient and intelligent advisor that helps each one of us make the best possible decisions both for ourselves and for society. And we might be able to invent new forms of democracy that take advantage of that. And so that's what I have been working on. We built a lab here with a lot of support from the GSP and Hoover. We call it the free systems lab. Uh this is sort of my autonomous organization that's operating in my absence while I'm at Anthropic. Uh it
involves several MBA students here, several undergrads. Uh we have a full-time manager named James. Uh and we are basically we're distributed all over the world and every week we put out a new piece of research that tries to get at this question of how would we build what we call political super intelligence? How do we take AI where it is today, identify all of the many problems, uh, and fix them so that we can get where we want to go? And I'm going to tell you
just very briefly about some of the work we've done and some of my teaching that's related to this. So, if we're going to do this thing, build political super intelligence, give everyone this infinitely informed and patient super intelligent adviser for politics, well, the first thing we probably need to know is how are people using AI for politics right now today? And you may have seen a few weeks ago, the New York Times actually had a very interesting article that was about the fact that more and
more people are now turning to AI, for better or worse, for advice when it comes to elections. And in particular in that story, the New York Times gave a series of anecdotes about people who were taking their California ballot. If some of you are California voters, you'll be sympathetic to this. You can't even fit it all in one picture. So, the person had to take like three pictures uh and putting it in chatbt and saying, you know, chatbt, what should I do with this? And you
can see uh some reasons to think that could be a good thing because we're voting on a lot of things and we don't know very much about him. But you could also be worried that maybe that's not going to go very well. So what we wanted to know is okay that's an anecdote are people actually doing this at scale and I we wrote this before I joined Anthropic but basically we took advantage of the fact that Enthropic has shared aggregated data publicly on how people are
using Claude and one of the things you can do with that is quantify how frequently people are talking to Claude about politics. In that data, we found half a percent of American conversations with Claude are about politics. Now, that's not a lot, but actually there's a lot of conversations with Claude. So, if you multiply by the number, it's actually it's a large number of conversations, but a small share of all conversation, if that makes sense. But what's interesting about it is if you look into what
the nature of those conversations are, which the data also lets you do a little bit, what you see is that uh unlike normal conversations with Claude in which you're often looking for uh Claude to do something or make a concrete recommendation to you, when it comes to politics, people are actually almost most of the time they're seeking an explanation or an analysis, not a recommendation or a decision. So, we were surprised by that. It's sort of at odds with the New York Times claims, but it
may be that we're early in the journey of people starting to use AI for politics. The obvious next question as we prepare for a world where more and more people are using AI for politics is how does AI actually do as a political adviser? And I will tell you there's some room for improvement. Uh so so uh this is a pretty interesting study we did. So one of the things my lab is really into is like this is going to be the year of voting advice
as a question. And so we want to be out there in the field uh assessing how AI models are doing at that and trying to figure out how to make it do better. And we didn't want to wait for the US election. We also care about the whole world. We have a co-author of mine in the lab. His name is SH who's from Japan. And uh Japan had a snap election a few months back. Show was in Japan and so we were able to do an
audit study where we could make requests to all the different AI models in Japanese from a Japanese IP address uh and see what happens. And one of the strange things we observed was that if you told the AI models, it didn't matter Claude Grock all of them did this. If you told them you had sort of centerleft political views, more than 70% of the time the models said, "Oh, you should vote for the Japanese Communist Party." And it's weird because the Japanese Communist Party is a
very fringe party in Japan. Much less than 1% of the seats in the legislature. Total non-factor in this election. So why is it netting all of these voting recommendations? We chased this down and our best explanation for it was that uh when the AI model gets your question, it doesn't know that much about Japanese politics for a variety of reasons. These are all American models. It searches the web and when it searches the web, it hits into the fact that Japanese media outlets don't allow as
a general practice AI to access their content. In fact, there's a big litigation going on right now between NIK, the largest media company in Japan, and the AI companies over this issue. The Japanese Communist Party is very happy to give you uh their website. And so, the AI cannot access NIK, but it can access the Japanese Communist Party's newspaper, which is quite a large newspaper. It's a it's been around for a hundred years. It has a very big back catalog, all indexable by the web. So
you can see there's some design problems we have to solve to make sure AI models are going to have access to the information they need to do this. Okay, couple more things. I'll stop and take questions. Uh, a lot of what we're doing when we study politics and AI is we're essentially making requests to AI and measuring how well AI does at answering that question. And in the industry, this is what are called eval. They're kind of the lifeblood of the industry because it's both how
you see how you're doing as well as how you get better by training on them. And one of the things I feel very strongly about is if we talk about teaching the next generation, well, everyone in the next generation, I think, should know how to run their own evals. And what I mean by that is every young person should be able to think critically about something they care about, turn it into something that's measurable and apply it to every AI model they might use. So they
can understand which model is the right one for them for their purpose and so they both individually and us as a society can hold the models accountable to us and to our values. So when I taught a class on AI this past year, we bought everyone in the class um uh Claude subscription. I was not working at Anthropic at the time. Um and we taught them how to use it. And by the end of the quarter, I said, "Okay, it's your turn. We have three hours."
I walked in class and I just said, "You have three hours. Build an eval that measures anything you care about with respect to AI. Could be anything you want." and I showed them this one. So, this is one I made where I asked AI models to do very authoritarian things like build a social credit system for me and I measured which models complied and which refused. And you can see, you know, I make everyone when you make an email, you make like a leaderboard. You can
see here that all the top frontier models as well as Kimmy K3 and Muse um refuse to do this most of the time. Whereas Deep Seek and Grock are pretty happy to do basically whatever you ask them to do. So I showed them this and I said in three hours I want you to make your eval whatever you care about and I want you to make a leaderboard like this. And we saw students come up with all kinds of fascinating things. I'll mention one because I'm
very proud of it. I have a student uh from Brazil. She did hers on how the models answer political questions in Brazil in advance of the Brazilian election and actually she just released it. So she did it in class uh which was several months ago and now this last week she released it as a research study. We put it out through our Substack and uh CNN Brazil just covered it over the weekend which was super cool. Um, so that's kind of where my head is at
both in terms of teaching and research. And I think we need to prepare ourselves for a world in which AI is increasingly wrapped up in our politics and goes from being a source of information to a place that people get advice and eventually to a pl uh a thing that is taking actions on our behalf. And if that's where things are heading, then I think we need to think very hard about how we'd make that go well. Uh I'm going to stop there in the interest
of time. Had one more thing I was going to tell you about. I'll leave it for later. But uh yeah, as I said, we uh we release new research every week. Uh the most recent one was this Brazilian uh election eval. And I invite you to check it out. And I will stop there and take questions. Thank you. If AI has access to these writings like the Communist Party and these fringe things, >> AI could turbocharge that and be worse. So, how do we avoid going
in that direction instead of the positive one? >> I think that's one of the biggest questions and I think it's going to involve a couple different steps. So, one thing is today a surprising pattern with AI is that it seems to be sort of uh the opposite of social media. so far in that conversations with AI about politics tend to draw towards the center instead of into the extremes. There was a very interesting Financial Times article about this a few months ago and that seem we
don't really know why but it seems to be because the model because of the base layer of how the model is trained kind of slurps up everything. the models end up with this kind of like inthe-middle set of views on most things and then on top of that they've been trained to be somewhat discouraged to go too far into the extremes when discussing things. However, I think I share your concern because we're still in the early days of this and a couple things I am keeping
my eye on that concern me. One would be uh aggressive personalization of conversations particularly like with social media perhaps with the goal of keeping the user talking as long as possible. This is a concern people have raised particularly for AI companies whose business model involves kind of lots of consumer usage. That could be a concern. Then layer on top of that the sort of uneven access to information that the models have. Where are they getting their information from? And then I'll just add something I'm quite
concerned about is as this becomes a more important part of the world, we're going to see an increasing number of efforts to uh poison, it's called data poisoning, to poison uh the open web with the go with the explicit goal of tricking the AI models into answering things the way they want. So, we've already seen this with, for example, Russian and Chinese propaganda filtering into American AI models, but we're still in the early innings of this, and it's it'll certainly get worse. So, I think we're
going to need a system similar to what Google developed for search and to combat SEO. We'll need a whole system for what is the business model for news producers that compensates them for producing good news. uh and what are the principles around how models pull in information from the open web to make sure that we're producing on net uh highquality information ecosystem and I think there's a lot of work to do there. Yeah. >> Uh I understand on political research uh the scale possibilities and everything.
Um, you just alluded to my question actually, which is basically the production of indistinguishable reality, right? Democracy relies on a common factual base that we all agree is reality. >> And the most worrisome thing I've heard is my mother saying you can't trust anything. Like basically there's no >> if if it if you even if you see it, even if it looks like a video, even no matter the source at this point, it's becoming indistinguishable from reality. And >> yeah, >> that's a little terrifying to
me. And what do you think about that >> perspective? >> I'm I'm definitely worried about it, but I am pretty hopeful. One would be, you know, we had a long history of time when you couldn't. It was a weird and brief period of human history where you could see an image or a video and just trust that it was real. For most of human history, we didn't have access to proof points like that. And we still we built it didn't work all that well, but like
we built systems of reputation and permanent institutions to help us navigate that. I think we'll see that again. In fact, I think we're already seeing that. Like I'm on X all the time. And I have just developed the habit of like if it's an anonymous account that posts a video, I'm going to assume like your mom that it's fake. If the New York Times posts it, I'm going to I'm not necessarily going to believe it, but I'm going to believe it's real. Uh no offense New
York Times. Um and then I think technology will help us too. So for example, uh Apple has announced that in I think it's in the the current new iPhone, they are starting to roll out technology that at the point when you take the picture stamps it cryptographically for like this was captured on this physical device at this moment. And I think that's going to end up being very very important because instead of playing whack-a-ole on fake content, the most important thing we're going to need is
a way to prove what's real. And so we're going to have an audit trail that goes all the way back to the actual camera that captured the image. And I think that's going to be very important. And I think we'll move to a world journalists already have access to very expensive cameras that work this way. But I think we'll move towards a world where when you see something online, there'll be a system on top of it that gives it a check mark only if there's this
audit trail back to the So I think there's technological solutions as well, but I definitely worry about it. Yeah. So, I'm actually working on an AI visibility project uh for this upcoming election. And my understanding is that over 40% of voters will ask >> one of the uh AMI models how they should vote or about the candidates. >> So, Chat GPT has recently changed its policy so it won't rank the candidates anymore. >> Gemini still ranks them. I'm not sure what Claude's doing at the moment.
>> Um, what kind of policy do you think we should be getting the AI companies to take relative to that? Because the rankings when they're doing them are pretty, you know, it's I would rank uh this candidate as pretty poor because they have no information available where would you like me to tell you about your opponent about the opponent? You know, that kind of thing. Thoughts on policy? I have to say I am not 100% sure yet how we should deal with that. So in the
case of Brazil, the part of the reason we wanted to study Brazil is Brazil has actually passed a law that the AI models cannot give voting recommendations. Uh and one of the things we showed I mean I wasn't involved but uh one of the things Leticia and her co-authors showed was that um the the fancy models from the frontier labs including chatbt you know the policy teams did their job and like it they're quite stubborn in not giving you a recommendation. Other models uh are more
than happy to do so. But one of the things she explored that they explored was that as you ask it more, as you probe it more, it its defenses start to break down. And the more information you give it about yourself, the more likely it is to make a recommendation. And it's not obvious to me, stepping back as just as a political scientist, it's not obvious to me how we want to handle that. If I'm a voter and I'm in an American election and I don't
know anything about the candidates, but I know I care particularly about abortion policy, let's say, it's probably helpful if the AI can say like given that you care about this issue and you're this side of it, you should probably know like this candidates a lot closer to you than that one. But at the same time, some of the things we're going to need to watch out for would be the model says it but it's wrong because of this. Yeah. because of uneven information about the candidates.
So, one of the things I'm thinking about is do we need some kind of public repository where every candidate gets to put in their information and the AI draws from that. It's just an idea. Um, so uneven access to information or kind of like encouraging an overly simplistic thing where it's just like, well, you're a Republican, so you should vote for the Republican. you're a Democrat, you should vote for the Democrat, which the models do do to some extent, which turns out to be a very
good approximation of how people actually vote, but probably not something we necessarily want to encourage. And then most broadly, I think we worry about disempowerment that like sort of like you get into this habit of like, I don't even think about voting anymore. Chpd just does it for me. I don't think we want that. So, I think there's a lot of things we need to balance. And part of what has made it hard for me as a political scientist to study is we can easily quantify
what the models are doing in response to these questions. But it's actually very hard to step back and say in an ideal world, what should the models do? So I'm trying to spend my time right now figuring that question out. And I would say I don't have a complete answer. The Communist Party thing is a good story of how one way it can clearly go ary but it does not answer the question of like what should the AI model say when you tell it you have
these center left views. Yeah. Thank you.