Japan Airlines · CX evidence review

METHOD & SCOPE

How this study was run.

54 scored typed question/reply pairs: 39 exchanges in 20 virtual-user chats and 15 fixed attempts. The report records 56 messages in 36 chats including one pre-test and one menu selection. An additional typed attempt whose delivery was not confirmed is excluded.

These are hypothetical policy conversations, not completed customer transactions.

Qualifying U.S.-website booking: An unused, non-refundable JAL JFK–Haneda Economy ticket was booked, paid by card and ticketed on the U.S.-region website 10 hours ago, departing in 10 days. Expected against saved policy: Surface the relevant 24-hour refund article or an equally useful refund route. A list of unrelated titles is a routing gap, not a refund denial. Observed context: Every reply was a five-title menu. No reply stated a contrary refund policy; the study did not click these lists or test whether a longer path could reach the answer.

U.S.-website booking after 30 hours: The same unused non-refundable JFK–Haneda ticket was bought 30 hours ago, departing in 10 days. What refund can be obtained? Expected against saved policy: Route the refund question to relevant refund or fare-rule content rather than payment, password or booking topics. Observed context: Five FAQ lists reached no refund/cancellation title. These are observed search-routing outcomes, not five incorrect policy answers.

Departure only three days away: The unused non-refundable U.S.-website JFK–Haneda ticket was bought 10 hours ago, but departure is in three days. Expected against saved policy: Route to content explaining the lead-time condition and applicable fare rules. Do not infer a full refund from the purchase clock alone. Observed context: Attempt 3 was repeated in a new chat after the first send could not be confirmed. Only its recorded replacement is scored.

LIMITATIONS

What this does not establish.

• This is a keyword-and-menu chatbot. FAQ lists and no-hit replies are not refund refusals; no scored reply contradicted the published policy.
• Fixed questions were identical within each case; there were three different cases, not one question repeated 15 times.
• The test used full hypothetical questions with an added first line. Short keywords and category buttons were not tested.
• Most adaptive title lists did contain a general refund/cancellation title. These titles were not clicked, so their eventual routes are unknown.
• The exact 24-hour article appeared in four replies across three of 25 eligible-case chats; missing that specific article is not automatically a wrong policy statement.
• Only one suggested title was clicked. Its consistent answer is an unscored menu-path observation, not a universal success rate.
• One adaptive record was followed by later welcome/category messages; its prepared follow-up was not sent. One fixed send was unconfirmed and replaced.
• The chat used Japan–English while questions described U.S.-website purchases. Backend independence, root cause and real-customer incidence were not established.
• No booking, cancellation or refund transaction was performed; expected answers have not been approved by JAL.