How accurate are AI symptom checkers in NZ, and can they affect insurance?

How accurate are AI symptom checkers?

AI symptom checkers are only modestly accurate. A 2022 review of 48 tools across 10 studies found that, in the nine studies that measured it, the correct condition was listed first 19% to 37.9% of the time. The authors called the accuracy variable and low.

In the six studies that measured it, advice on how urgently to seek care was right 49% to 90% of the time. These studies mostly did not verify vendors' AI claims.

A second review a year later reached the same conclusion from a different set of studies, describing diagnostic accuracy as low overall.

Accuracy varies a lot from tool to tool

Accuracy varies sharply between AI symptom checkers, so a figure quoted for one tool tells you little about the next.

Every study in the 2022 review that tested more than one tool found marked variability on the same patient cases. A 2024 study reached the same conclusion, finding the variation substantial enough that symptom checkers cannot be treated as a single entity.

How that compares with a doctor

Doctors outperform symptom checkers on the same cases. In one study, 234 physicians and 23 symptom checkers worked through the same 45 patient vignettes. The doctors named the correct diagnosis first 72.1% of the time, against 34.0% for the tools.

The size of the gap depends on which measure you use, and vignettes are not real patients. The direction is consistent, though. In 30 orthopaedic cases, physicians judged urgency correctly 70.0% of the time, compared with roughly 21% for an AI apps.

How does an AI symptom checker work?

An AI symptom checker works in three steps. You enter your main symptoms, then answer follow-up questions that fill in the details. It returns a list of conditions that might fit, with advice on how urgently to seek care.

It works without things a clinician would typically have: no vital signs, no examination, no medication list, no test results. What comes back is a shortlist, not a verdict, and it can miss something important because it only knows what you tell it.

AI-powered checkers vs older symptom-checker tools

Older symptom checkers followed fixed rules. They worked through a decision tree of predefined questions and returned whatever the tree pointed to. AI-powered ones use AI instead of fixed rules, which lets them interpret what you type more flexibly and read more context.

Both kinds work only from what you type, and neither examines you. The rest of this article uses 'AI symptom checker' to mean the model-driven sort. The reviews behind the accuracy evidence above and below did not verify vendors' AI claims, so that evidence cannot be cleanly split by tool type.

What "accurate" actually means for a symptom checker

'Accurate' means at least two different things here, and they are measured separately. One is whether the tool names the right condition. The other is whether it tells you to seek care at the right level of urgency. A tool can do well on one and badly on the other.

That is how two studies can both be right and still disagree about how good a tool is. A headline accuracy figure tells you little on its own.

Getting the condition right vs getting the urgency right

A tool can be poor at naming your condition and still useful if it tells you to see someone today. Researchers argue that triage accuracy matters more than diagnostic accuracy, because these tools increasingly work as triage services.

Triage can fail in two directions, each with a different consequence. Over-triage sends you to care you did not need. Under-triage tells you to stay home when you should not, which is the one that delays treatment. The review finds both, and calls under-triage the concerning one.

Why one tool can look good and bad in the same study

A tool's score changes with how strictly you count a hit. Counting only the top answer, nine studies found symptom checkers named the right condition 19% to 37.9% of the time. Counting anywhere in the first three, seven studies gave 33% to 58%. Separately, this review finds a tool can score well on urgency and poorly on the condition itself, in the same study.

Where AI symptom checkers get it wrong

AI symptom checkers are weakest where you would most want them to be strong: uncommon conditions, complicated cases, and situations missing key context. In one audit, the right diagnosis came first for 38% of common conditions but only 28% of uncommon ones.

Three patterns recur:

• Rare and uncommon conditions. One study reviewed in a 2022 systematic review found infectious diseases were named correctly only 3% to 16% of the time.
• Complex and overlapping problems. Tools may be less reliable if you have several health problems or take several medicines.
• Context they never ask for. One user described a tool that could not get near the right answer without asking about their race.

AI symptom checkers are also weakest at spotting emergencies. A comparison five years apart found the typical app's ability to spot an emergency was lower in 2020 (51.9%) than in 2015 (85.7%). A tool that misses an emergency can give false reassurance instead of a warning to seek care. Blindly trusting a tool's result carries its own risk, and that risk is greatest for people with serious conditions.

Who regulates AI symptom checkers in New Zealand?

There is no pre-market approval step for medical devices in New Zealand, and a 2025 Cabinet paper concedes the Medicines Act does not adequately regulate software as a medical device. Medsafe regulates medical devices and states there is no approval system for them under the Medicines Act 1981.

Suppliers must notify devices to the WAND database, but Medsafe says notification does not mean a device has been assessed for quality, safety, efficacy, or performance. You cannot look a tool up either, because access is restricted to sponsors.

The same Cabinet paper states that manufacturers face no requirements on quality, privacy, or cyber-security. A replacement Medical Products Bill is being drafted, with changes reported in 2026 as not taking effect until 2028.

Health New Zealand is separately assessing a symptom checker of its own, with a governance group covering clinical safety, ethics, and equity. That work is still under way.

How to choose an AI symptom checker

It's worth choosing an AI symptom checker the way you would choose any health app. A few things are worth checking before you trust one, starting with who built it and what evidence it rests on. Tools from reputable organisations are usually built with clinicians involved.

• What it is built on. Information from medical institutions, health professionals, or peer-reviewed studies is the more reliable kind.
• Who reviews it. Partnerships with health organisations, clinicians, or universities are a reasonable proxy for accuracy.
• What happens to your data. Check how it is collected, stored, and used, and whether it is shared or sold. A privacy policy alone does not mean it stays private.
• Whether it knows when to stop. A good tool may tell you to seek help rather than leaving the call to you.
• Whether it was written for New Zealand. Many are made overseas and may carry information that differs from New Zealand recommendations.

If you would rather talk to a person, Healthline is free 24 hours a day on 0800 611 116. In a life-threatening emergency, call 111.

How to use an AI symptom checker safely

An AI symptom checker's result is best treated as a starting point, not an answer. Health New Zealand says the tool it is assessing does not replace clinical judgement, which stays with healthcare providers.

A few habits make these tools safer to use:

• Answer honestly, because the result only reflects what you put in.
• Use the result to prepare for an appointment, writing down your symptoms and how long you have had them.
• Check what it tells you with a clinician, and it's best not to change your medicines or treatment on an app's say-so.
• If your symptoms are getting worse, stop checking and seek help.

Some symptoms should skip the app entirely. Call 111 immediately if you notice heart attack warning signs. For stroke, the FAST signs mean calling 111 rather than waiting to see whether it passes. With a child, you know your child best, and a child who is floppy, blue around the mouth, or unresponsive needs urgent help.

Could using an AI symptom checker affect your insurance application?

What an insurer asks about is your health: the symptoms you noticed and the advice or treatment you sought. Because disclosure turns on what you knew rather than how you looked into it, using an AI symptom checker is unlikely by itself to affect your insurance application. The tool itself is unlikely to be what is being asked about.

That reasoning rests on two sourced points. A symptom can be a pre-existing condition even without a specific diagnosis. And an AI tool does not produce a diagnosis in the first place. What you carry into an application is the symptom, not the screen.

An AI result is not a diagnosis

Healthify says AI cannot safely or accurately diagnose a medical condition. It does not have the full picture of your health, so its answers should not be used to diagnose or rule anything out. Used with care, these tools can help you prepare for a doctor's visit. What you have is a list of possibilities, and only a clinician can turn that into a diagnosis.

What you need to tell an insurer

It's the symptoms themselves that need to go to your insurer. Under current New Zealand law, you're required to tell an insurer anything that could affect its decision to insure you. Information counts if it would have influenced the judgement of a prudent insurer. A symptom counts even with no diagnosis, no tests, and no treatment.

This guidance is for individually underwritten cover. Some employer-arranged group cover uses automatic acceptance with no individual health questions, often up to a scheme limit, so the disclosure answer can be different there.

That means covering:

• symptoms you noticed, however minor
• advice you sought, including a GP visit about something that then settled
• tests, referrals, and treatment, and what they found

Typing symptoms into an app is not the same as seeing a doctor about them. The symptom that sent you looking is still yours to disclose.

The duty often continues after you apply. For life, income protection and health cover, you're generally required to tell your insurer about anything that happens between applying and cover starting.

Having pre-existing conditions doesn't mean you can't get cover at all, or that these health issues will be automatically excluded from your policy. Some policies exclude a pre-existing condition at enrolment.

Others cover it after a set period, and some may allow cover after a scheduled review, though that isn't guaranteed. A few carry no pre-existing exclusion at all. The details vary by policy. There is more in Policywise's guides to health insurance and pre-existing conditions and life insurance and pre-existing conditions.

What happens at claim time if you didn't mention it

If you did not disclose something material on individually underwritten cover, the insurer can avoid the policy back to the beginning under current law. The policy is then treated as though it never existed. That can surface years later, when you claim, and the current law does not distinguish an honest mistake from a deliberate one.

One published case shows the shape of it. A claim was declined on a pre-existing knee condition after the insurer found records noting degenerative changes 15 years earlier.

Insurers sometimes do something less drastic, applying the terms they would have offered had they known: less cover, a higher premium, or an exclusion.

If you cannot remember your history, it's a good idea to ask your doctor for your notes before answering. Insurers do not automatically ask your GP. If a decision goes against you, your insurer's complaints process comes first, and the Insurance & Financial Services Ombudsman Scheme can then investigate for free.

The law is changing, though not yet. The Contracts of Insurance Act 2024 will stop insurers cancelling a contract simply because a consumer failed to disclose something, and will require proportionate responses. It is not yet in force.

An adviser can help you get this right. Policywise can tell you what belongs on a health insurance or life insurance application before you sign it, and stays involved at claim time.

RECOMMENDED READINGS

Will AI replace insurance brokers in NZ? What AI can and can't do for your cover

Can you trust AI chatbots for insurance questions? Benefits, risks, limits, and when to see a licensed adviser

How AI is changing health insurance underwriting and claims in NZ

Not sure how your health history affects your cover? Let Policywise help!

Policywise is a 100% free service which tells you which health, life, and disability insurance provider best fits your needs. We offer fast, comprehensive, and easy-to-understand comparisons of all leading providers, and a simple summary clearly recommending which insurer is best for your situation.

Not all insurance policies are the same. Policywise can help you sort out the duds, avoid the lemons, understand the fine print and exclusions, and get the right insurance for you and your family.

We make the important decision of where to buy your insurance super easy. We’ll answer your questions, provide experienced advice and quotes, and manage all the back and forth throughout the application process. Taking out your cover through us means you'll have our lifetime support and claims advocacy, and we'll help you negotiate a positive outcome at claim time. We can also take care of lodging any claims on your behalf and back you up if the going gets tough.

Check out the reviews on our homepage for how other New Zealanders have found our service, because now is the time to get your personal, family, or business insurance sorted. Give your family or someone you love the most outstanding financial support possible. Book a 5-minute callback with Policywise today; our service is fast and free.