If you’re on social media, you’ve likely seen many of the uses for ChatGPT as a travel planning tool. To the naked eye, it looks great but it’s a façade that sets users up for failure.
ChatGPT (and other AI tools) Can Plan Everything For You, Right?
There are a ton of social media examples demonstrating that AI can do everything under the sun but there are some obvious things it cannot. I should be clear, I run a business outside of the travel industry that uses ChatGPT every single day. We’ve developed our own tools, tried a ton that are already on the market, and have a pretty good grasp on what is and is not possible.
When I see videos like this one (and this is just one of many I have seen this week), it’s a little frustrating for some obvious reasons. It shows what we hope to get from GPT but it’s incomplete at best. Here’s an example:
View this post on Instagram
AskAI is an app that uses GPT and markets its ability to help with travel planning. It was advertised to me (I have travel interest, obviously) in social media so I downloaded it to see what it could do.
I tried the prompt: “Plan a trip for a family of four with activities to a resort in Jamaica for a week in May with costs and distance from the resort.”
Here’s what it returned:
You only get three questions through this app without paying and the output can’t be exported without a paid membership.
Where It Succeeds
For general information, these tools are pretty good. Not perfect, no, but pretty good as they are today. All of the places in the above example are real places, they are generally true statements even if slightly imperfect.
For example, using tools like AskAI or ChatGPT as the video suggested would help someone to outline a possible itinerary of generalized places that could be close to a hotel or point of interest approximated.
The ability to output results as a spreadsheet as the Instagram example demonstrated is helpful, especially for the analytical, and adds to the utility of AI for trip planning.
Where It Fails
The key component that needs to be understood (but isn’t by most) is that this data set was last crawled in September of 2021. That’s really important for a few reasons:
- Nothing that’s happened after that point is in the data set of GPT, but some of its collection would have happened far earlier.
- The data set is weighted by information available at the time of collection which can skew from later accepted knowledge.
- There’s no communication between the GPT model and the internet so it has no access to current information.
This has a few real-world consequences.
First, prices are inaccurate almost universally. In my research, GPT not only invented a ticketing category for the Eiffel Tower that doesn’t exist, but all of the prices were inaccurate. I spoke early on about AI writing here. We see this too in the example from above where GPT has read the price range to be “per person” when it was actually priced for two people. A weeklong stay, according to GPT through AskAI would be $4,000-8,000 for the hotel mentioned. I shopped random weeks in May and found the price in the middle of those two at $6,300 for two people.
That’s not always the case, however. Results for “Paris hotel in July under $300” was wildly inaccurate.
It’s also going to be skewed by voluminous content. Places that have been extensively written about will weigh more than those with less digital ink spilled. I conduct the same test example on any GPT tool I try, which is to ask the AI to write me a post about the attacks on September 11th. Many tools return with common phrases like, “It’s been ten years since the attacks of September 11th” or “Now, a decade since the attacks” which is likely because so much content was generated about the topic on the tenth anniversary. If you ask GPT how many years it has been since September 11th, it will know that it’s been about two decades because it knows when the attacks occurred. But if you just ask it to fill in content, that’s what many tools will output.
Why does that matter for trip planning applications? Because if a resort had tons of glowing reviews 15 years ago and relatively few since then, it will believe the majority of the content is good, not that it’s gone downhill since then. If more content was available about the moon landings being faked in 1969, rather than taking place, GPT may skew toward the former rather than the latter.
Marketing guru, Gary Vanyerchuk, was asked at a conference how to use AI by a realtor. He responded that you could instruct AI to write LinkedIn posts comparing mortgage rates from 100 years ago to today. That’s the crux of this issue. GPT has no idea what mortgage rates are today and from the time the information would have been gathered it would have differed wildly from what is on the market now due to adjustments in the fed. Here’s what GPT says,
“Mortgage rates 100 years ago were significantly higher than mortgage rates today. In 1920, mortgage rates averaged around 6.5%. Today, the average rate is around 3.5%. This is largely due to technological advances and the introduction of the Federal Reserve System, which has helped to stabilize interest rates.”
In reality, the average market rate today is 6.8%. With the average home price around $300,000, that’s a payment of $1,956/month compared with $1,347 if the rate was still 3.5% – a difference of nearly 50% higher now rather than what GPT said. If someone followed Gary V’s advice, they’d look like an idiot on LinkedIn when they intend to portray themselves as the expert. He knows better than to suggest that and anyone recommending similar advice that involves current pricing availability, or even routes should know that too, and caution against it.
Factoring In The Future Before It’s Happened
I debated with another blogger this week about some of these issues and his suggestion was that it’s more important to what the tool will be able to do in five years rather than what it can do today. Many share this view and as he pointed out that GPT5 (two and a half iterations from the initial release) could be out inside of a year of the public launch. (OpenAI CEO, Sam Altman, spoke at MIT and announced that it’s no longer working on GPT5.)
That’s incredible growth. However, we can’t factor in what hasn’t happened yet. Versions GPT3, 3.5, and 4 have all relied on the same crawl of data so while it may get better at speaking and recognizing communications – after all, it is a language model – it doesn’t have any better grasps on what has actually happened since it last crawled.
Consider what the world looked like in September of 2021. The world was still grappling with COVID and shutdowns, Russia had not yet invaded Ukraine, and just days prior the US withdrew troops from Afghanistan. It seems so long ago now that we were all masking up on planes and no one was traveling. The fastest way from Helsinki to Moscow varies entirely from then to now.
It’s hard to understand what future iterations will or will not bring, and while my colleague and I agree that the expansion of the tool has been a hockey stick growth pattern, we differ on how long that will extend and the total impact. We also disagree on the use of speculative tools rather than precise ones; speculative tools have been proven to be inaccurate, sometimes wild fabrications.
In the future, it’s possible that AI will be able to check hotel availability in real-time, providing accurate prices and occupancy and complete everything but actually taking the trip. That very well could happen. But it doesn’t happen now and it may never happen.
GPT and AI are great tools for certain applications. Things that have happened in the past that will not change and do not need to be current are where it succeeds. However, what could happen has not yet happened, and failing to evaluate the results could have real-world consequences. Before using AI to help you plan your trip, consider the limitations of AI, and if you use it, verify the information is correct and current still as it has likely changed.
What do you think? Do you use AI for trip planning? Have you run into struggles?