_____ Agentic travel experiment

AI is becoming an increasingly common consideration in how people plan their holidays. According to Booking.com's global AI sentiment report, 67% of consumers surveyed had already used AI in some aspect of travel, and 89% said they wanted to use AI when planning future trips. 

The most common use of AI during travel planning was researching destinations and the best time to visit, as cited by 38% of respondents. The report also found that 24% trusted AI assistants as a source for travel planning, ahead of colleagues (19%) and influencers (14%).

With more people using AI to research holidays and turning to it for recommendations, we wanted to explore whether AI assistants approach holiday research in the same way as people, and how their recommendations may differ.

To investigate, we gave seven humans and two AI models (ChatGPT and Gemini)  the same in-depth family holiday brief, with each AI model completing 100 independent runs. We then compared their research journeys, looking at the websites consulted, which hotels they ultimately recommended and how closely the recommendations matched the original requirements.

Key findings from our experiment 

The brief was to find a seven-night family holiday to Greece, with flights from London and a maximum budget of £4,000 for two adults and two children. Here's what we found.

Our hypothesis 

When researching holidays, people often draw on their previous experiences with travel operators, alongside recommendations and reviews from other travellers. Someone who has booked with TUI before, for example, might naturally start their search there, while another person may prefer to compare hotels across several booking websites.

AI assistants can approach this process differently, using information retrieved from online sources to identify and recommend suitable hotels. We were interested in whether the websites they relied on would lead them towards the same recommendations as our human researchers.

Based on this, our hypothesis was that:

We also wanted to see how reliably AI could follow the booking requirements, particularly when it came to budget and the number of people travelling. Given the level of detail in the brief supplied, we anticipated the AI assistants to be capable of meeting these requirements as reliably as our human researchers. 

Our methodology 

When coming up with the brief, we wanted it to be a realistic family holiday request, with enough detail to help us assess whether the recommendations met the requirements. 

Every human participant and AI run received the following brief, word-for-word: 

How we conducted the experiment 

We divided the experiment into three research groups, with the human participants acting as our control group. 

Human researchers

Our seven human researchers were asked to research the holiday without using any AI assistance. They used an in-depth travel log for each person to record their searches, websites visited, hotels considered and prices they found.

Once the research concluded, the human researchers recorded their final hotel choice, rated their confidence in how well it satisfied the outlined brief and whether they’d book the holiday themselves.

AI assistants 

For the AI research, we tested two models - GPT-5.6 Luna and Gemini 3.5 Flash Lite - using the same holiday brief. Each model completed 100 independent runs, giving us 200 AI research journeys to compare how consistently hotels and websites appeared in their responses.

ChatGPT’s research logs included its search queries, retrieved results, pages opened and links cited in each response. This level of detail meant we could compare websites ChatGPT found during the research process with those it chose to reference in its final answer.

Gemini’s API provided us with less detail about its search queries, so we recorded the queries, hotel recommendations and sources cited in each response. We also followed the redirect links during each run to identify the original websites behind its citations. 

The cost of running the AI research in this study was approximately $4 per 100 AI runs, working out at around 4 cents per run. 

What we tracked

Precautions and limitations

To keep the experiment as consistent as possible, every participant and AI run received the exact same holiday brief. Human researchers were instructed not to use AI tools when researching, including AI Overviews in Google Search, and were asked to record any AI-generated results they encountered.

We also checked which travellers the quoted holiday prices covered and spot-checked frequently recommended AI hotels to confirm they existed.

There are a few limitations to consider when interpreting the results of this study:

The findings of our experiment specifically look at the behaviour of these two AI models when given the same holiday brief, as opposed to how every AI assistant would approach holiday research tasks.  

The results: How did humans and AI research the same holiday?

Our findings largely supported our original hypothesis, with the human participants and AI models taking noticeably different approaches to researching the same holiday.

The sources they used varied considerably, as did the hotels they eventually recommended. We also found differences between ChatGPT and Gemini, particularly in how they approached the research and how accurately they interpreted the booking requirements.

1. Humans relied on familiar travel brands and reviews, while AI favoured affiliate content

The websites used to research holidays differed considerably between the human participants and AI assistants.

Top websites visited by humans and cited by AI assistants

Six of our seven human researchers visited tui.co.uk’s website, while all seven checked review websites before making their final recommendation. One participant specifically mentioned having booked with TUI previously, which influenced their trust in the operator and later their hotel of choice.

ChatGPT, meanwhile, cited freechildplacesholidays.co.uk in 91 of its 100 runs. Gemini's sources were more varied, with Mumsnet appearing in 46 responses, followed by kiddieholidays.co.uk (39) and bestchildfriendlyholidays.co.uk (38).

Looking at the types of websites used, just one of the seven human researchers consulted a blog or editorial source, compared with 91% of ChatGPT runs and 71% of Gemini runs citing this type of content.

The opposite was true for review websites. All seven human participants checked sites, like Tripadvisor, but only eight ChatGPT runs linked to a review website. Gemini didn't cite one in any of its final answers, despite mentioning Tripadvisor in 49 responses.

We also found differences between the websites ChatGPT encountered during its research and those it ultimately cited. Reddit appeared in search results in 99 of the 100 runs, but wasn't cited once. Social media sources accounted for 24% of ChatGPT's search results, yet none of its final answer links pointed to them.

Which types of websites featured most frequently?

Beyond individual websites, the table below shows how often different source categories appeared across the three research groups.

The biggest differences between the AI models were in their use of online travel agents and tour operators. Gemini recorded 107 cited appearances from online travel agents, compared with 26 for ChatGPT, while ChatGPT recorded more citations from tour operators (105 versus 69).

2. ChatGPT recommended the same hotel 76 times, despite no human considering it

Across their seven research journeys, the human participants considered 25 different hotels. However, not one considered TUI BLUE Elounda Breeze in Crete, which ChatGPT recommended in 76 of its 100 runs.

In fact, ChatGPT's recommendations were heavily concentrated around just three hotels, which accounted for 85% of its final choices, compared with 38% for Gemini's three most frequently recommended hotels.

Gemini produced a much more varied selection, with its most frequent recommendation being Atlantica Marmari Beach in Kos, appearing as the final choice in 17 of its 100 runs. This was also the final choice of one human researcher and appeared once as ChatGPT's final recommendation. Another human choice, TUI SUNEO Tsilivi Admiral, was recommended by ChatGPT in just two runs.

Which hotels did ChatGPT and Gemini recommend most often?

There were disparities among the researchers and AI assistants when it came to which destinations in Greece they recommended. 

Which Greek destinations did each group choose?

None of the seven human researchers recorded a final choice in Crete, although five of the hotels they considered were located there. Their final choices were instead spread across Rhodes, Kos, Zante, Kefalonia and mainland Parga, with one destination unrecorded.

ChatGPT, meanwhile, recommended a hotel in Crete in 80 of its 100 runs. Gemini's recommendations were more evenly distributed, with Crete appearing in 23 runs, followed closely by Kos and Rhodes (21 each).

3. AI researched holidays in seconds, but the results differed considerably between models

One of the biggest differences in our study was how long the human researchers and AI assistants took to complete the same holiday brief. Our human research journeys averaged approximately 57 minutes each, compared with just 26 seconds for ChatGPT and seven seconds for Gemini.

How long did the research take?

ChatGPT averaged 10.5 search queries per run, compared with 2.5 for Gemini. It also retrieved approximately 70 search results per run, often searching specific travel operator websites, including TUI and Jet2holidays.

Gemini averaged fewer search queries and typically searched more broadly for family-friendly hotels in Greece, although its final answers named more hotels and cited a wider range of websites than ChatGPT. 

How did ChatGPT and Gemini's answers differ?

The differences between the two models became more apparent when we looked at their final responses. Across 100 runs each, Gemini named 73 different hotels, compared with 53 for ChatGPT. It also linked to 69 different websites, almost twice as many as ChatGPT's 39.

Gemini's responses contained more links overall, with 1,236 across its 100 runs compared with 619 for ChatGPT. However, the number of links alone doesn't necessarily indicate how thoroughly each model researched the holiday, particularly as their search tools provided different levels of detail.

*Based on 100 runs per model. Hotels named includes all properties mentioned in the responses, while final selections include broader recommendations and answers without a specific hotel.

Despite including more hotels and links, Gemini was less likely to settle on one clear hotel recommendation. ChatGPT identified a single final choice in 91% of runs, compared with 72% for Gemini. This suggests that the two models differed in how they presented the results of their research, with Gemini more likely to leave travellers with multiple options to consider.

How closely did AI follow the holiday brief?

There were differences in how closely the AI assistants followed the original booking requirements. ChatGPT identified travel dates within the requested August window in 97% of runs, compared with 71% for Gemini. In 21 Gemini runs, the answer provided a Greek arrival airport where a London departure airport was expected.

Gemini also cited websites outside the UK, including thetravelexpert.ie in 35 runs and expedia.ca in nine, despite the brief specifying flights departing from London. It's worth noting that Gemini's search tool didn't support the same location controls as ChatGPT's, which may have contributed to these differences.

The models also handled the instruction to book differently, with ChatGPT acknowledging that it couldn't complete a booking in 99% of runs, compared with just 1% for Gemini.

What influenced the human researchers' choices?

The human research logs showed how much personal preference influenced people's final choices. Some of the participants focused on how the hotels looked and the facilities available for their children, with one specifically stating they were looking for waterslides. Another favoured TUI because they’d booked with the operator before and trusted the brand.

Children's facilities also featured heavily in the AI recommendations, with both ChatGPT and Gemini stating that their chosen hotels had a kids' club in approximately 88% of runs. However, these claims weren't always supported by information from the hotels themselves.

Three of the seven human researchers also checked maps or Street View to get a better idea of the hotels' locations, while another three used TikTok or YouTube during their research.

Finding all the information needed to compare holidays wasn't always easy, especially when it came to flight details and the overall cost. One human researcher noted that “you have to go through to the booking process before seeing what flight times are available”, while another struggled to establish the total cost of the holiday when prices were displayed per person.

Despite spending an average of 57 minutes researching, the human participants weren't convinced by the holidays they'd chosen. When asked whether they'd actually book their chosen holiday, only one said yes - although the group gave an average confidence score of 3.8 out of 5 overall.

4. AI holiday prices appeared cheaper, but didn't always cover the whole family

Pricing was one of the biggest differences between the human and AI research journeys, especially when it came to establishing the cost of a holiday for two adults and two children in Greece during the summer holidays.

All seven human researchers recorded package prices for the family of four, including flights. Their final choices ranged from £3,351 to £4,600, with an average of £3,938. Three of the seven selected holidays exceeded the original £4,000 budget.

In comparison, ChatGPT quoted a price of £2,894 for TUI BLUE Elounda Breeze in 75 of its 100 runs, approximately £400 cheaper than the lowest package price (£3,311) recorded anywhere in the human research.

However, the £2,894 figure came from an affiliate listing priced for two adults and one child, and not the family of four specified in the brief. Of the 75 runs quoting this price, ChatGPT presented it as covering all four travellers in 35, correctly identified the smaller party size in 25 and didn't specify in the remaining 15.

In 40 runs, ChatGPT went a step further and told the researcher that they would have £1,106 left over from their £4,000 budget, even though the quoted price didn't include a second child.

How did AI assistants present holiday prices?

Gemini approached pricing differently, providing a single total price in just 13 of its 100 runs, while 71 gave a price range. The most common was £3,200 to £3,800, although these were estimates rather than confirmed package quotes for the requested dates and family size.

This was quite different from the human research, where three of the seven participants selected holidays above the £4,000 budget. One participant even described the budget as “crazy low for this trip”, highlighting some of the difficulties people encountered when trying to find a family holiday that met all the requirements.

How did the holiday prices compare?

To recap

Our experiment largely supported our original hypothesis, with the human researchers and AI assistants relying on different websites and arriving at very different hotel recommendations, despite receiving the same holiday brief.

The results also challenged our expectation that AI would be at least as reliable as humans when following specific booking requirements. ChatGPT repeatedly quoted a price that didn't cover the whole family, while Gemini was less likely to provide specific dates or a clear hotel recommendation.

Although our research involved just seven human journeys and 100 runs of each AI model, the differences highlight some important considerations for travel brands, particularly around which properties are being recommended and the information AI assistants are sharing about them.

What should travel brands take from these findings?

For travel operators, online travel agents and hotel groups, there are several practical checks worth considering, including the following.

Check which third-party websites AI assistants are citing

Our results showed how heavily ChatGPT relied on one affiliate website, while Gemini used a different mix of sources. Reviewing the websites cited for relevant holiday searches can help identify where your hotels appear in third-party content and which sources may warrant closer attention.

Track which individual hotels are being recommended

ChatGPT recommended the same hotel in 76% of runs, despite none of our human researchers considering it. Monitoring non-branded holiday prompts can help establish which properties AI assistants favour and whether those recommendations change across different platforms.

Check how accurately your holiday packages are being presented

In 75 ChatGPT runs, the assistant quoted a £2,894 holiday price that covered three people, despite the brief specifying four. Travel brands should check whether AI-generated recommendations reflect the correct family size, travel dates and package inclusions, particularly when prices are being presented as confirmed totals.

Review whether AI assistants can access the information travellers need

Our human participants considered practical details such as luggage allowances and facilities for children, while the AI models didn't always provide property-specific evidence for their claims. Checking what information is available and how AI assistants use it could help identify gaps in hotel and package content.