Why did THIS movie get picked?

Why did THIS movie get picked?

Is it reasonable to expect over half of the people in any given group pick the same somewhat random movie? 

Something doesn’t smell right. 

Here’s the scenario:

There were about 750 copywriters back in spring of 2023, participating in a AI challenge using ChatGPT. We were trying different prompts and trying out different ways to use AI. 

Some were funny.

Some were silly.

Some were worthless.

A couple were beneficial.

But we all noticed a few concerning things.

One of the main problems that came up was the limited pool of data used.

Before I get into the specifics, we have to get a little technical and give just a little MORE background. 

What is this thing we call AI?

Keep in mind that what we call AI (artificial intelligence) isn’t intelligent the way you may immediately think. It isn’t really processing information the way you or I would. It is merely making an assumption about what is the most likely the next word to be used. It is literally called “hallucinating.”

ChatGPT has an overall general knowledge base. If you ask it, it will say something along the lines of: I have a foundational knowledge set that includes broad information about various topics, like history, science, literature, culture, and much more, up to my knowledge cutoff in October 2023. This general data covers factual information, explanations, summaries, and other widely-known or established information. 

When I asked for more specifics about what kind of general information, it said it included, “Cultural Knowledge: Information on movies, music, art, and literature spanning different time periods and regions. I have summaries and insights into popular works, influential figures, historical movements, and general trends.”

That was interesting to me because my own experience was despite the fact that it has ALL KINDS of information, it spit out a NARROW data set as an answer. 

There were about 30 of us who did one particular exercise and reported back and commented about our results.

The prompt we used started like this, “You are a film expert who specializes in making recommendations people love. Please recommend 5 movies I should watch this week, based on my own unique preferences and personality…”  

Then we had a specific prompt template about how to say who are and what kind of movies we like. 

The two things that surprised me the MOST were that the results were NOT recommendations of Academy Award winners or blockbuster movies and the REPETITION of the same movies. 

The other thing that became clear was the limitation of how the model makes decisions (more on that later and how that has changed). 

So, what do I mean? 

The two movies that kept showing up again and again were two 2013 movies: The Secret Life of Walter Mitty and About Time.

Regardless of what the personality type was of the individual requesting movies, if they requested any kind of romantic comedy or adventure, one or the other would be a recommendation. 

Obviously, there is some kind of choice architecture happening behind the scenes to cause that decision-making to happen. 

One of the copywriters participating in the challenge said that she filled every question in on the prompts and even added more specific personal details and still got the same results. Even when she asked for different movies, she didn’t end up with anything that wasn’t on anyone else’s list.

Here’s what I said at the time:
It seems to me there was a VERY small pool of movies that popped up again and again regardless of the diversity of the individuals who prompted the requests. WIth the vast number of movies produced over the last hundred years, there should have been enough movies to give us each a unique set. Or even a random sampling from the last couple of decades. And, yes, very US-mainstream oriented. Not quite blockbusters – I live in a smaller town and several of the movies never even played here – but I’ve heard of all of them (and I’m not a big movie-goer).

One of the other copywriters commented, “I think with the movie lists, that’s where you see a big bias (doesn’t India actually produce more movies?). I really had to urge it to mention European output. Earlier I’d already said I’m not a fan of major Hollywood schmaltz, but it still listed it!

But where does that leave us?

Here are some of the challenges:

If ChatGPT gives something (a movie, restaurant, etc.) as a “top” recommendation, more and more people will go there/watch it. As more people see it, comment on it, talk about it, it will become more popular and then that lends more weight for it being the “best” answer. It becomes circular. 

And harder and harder for a new option to break into the category. 

Maybe this is okay for movie reviews, but can you imagine the chaos as more people turn to AI for their search engine? How does a new shop compete? Or one that was mediocre and gets a new chef? How do they ever gain a footing when there is an echo chamber of reviews about what is good?

Another challenge is that we tend to think of data as being neutral. Facts are facts. But we now live in an environment where we have TONS of information at our fingertips. We don’t have to be discerning about where it all came from. We just throw out the question and get the answer. But when your answer is curated from sources you didn’t select, the facts you get back aren’t neutral. They have a worldview you didn’t pick. Maybe one you disagree with but you don’t even notice. Maybe it pulled data from an uncreditable source or from a source you philosophically disagree with so you would disagree with the outcome. 

The other problem is LLM writes literally and without any emtion or context. But we read through the lens of our human experience. And we have a shared humanity so I don’t need to explicitly write the whole context for someone to know where something is happening or what is going on. Our shared knowledge fills in the blanks. Using AI, it doesn’t know what to add or leave out.

And there’s a whole legal discussion around copyright that we can get into some other time. 

So use the AI Tools available to you. But use them wisely. Don’t turn off your brain. Tell it what you want it to work with and how you want to see the results. The more you give directions, the better the output will be. Don’t just cut and paste the results without reviewing it. Edit it. Reread it with fresh eyes just like you would any other copy. 

One more note: I think if you tried the same challenge & prompts today, you MAY get a different answer. As the model learns what works, it is going to give better answers. But you are still going to get the BEST results when YOU feed in the data you want the system to use. The more work you do, the better results you’ll get.