Our analytics counted US visitors twice every evening

We build Privatrak, a cookieless analytics tool. In September one of our testers, who lives in the US, told us something was off. His data looked weird: there were more visits every day than there should have been.

He was right. For visitors in the US we were counting some people twice, every day, around dinner time. It’s obvious in hindsight and was invisible before: we had never decided what a “day” is.

How we count people without a cookie

A cookie is a label stuck on a browser. Every page view carries it, so counting unique visitors is easy.

We don’t store anything on the visitor’s device. Instead we compute an identifier on the server from things every request already contains:

visitor_id = HMAC(daily_key, IP address + browser + site)

The same person on the same device produces the same identifier on every page view, so we count them once. We never keep the raw IP address or browser string, only the hash.

The daily_key is what keeps this private. If it never changed, the identifier would be a stable fingerprint that follows someone for months, which is exactly what cookie rules exist to prevent. So the key changes once a day, and after the change yesterday’s identifiers can’t be matched to today’s.

What we hadn’t thought about: “once a day” means “at some clock time”. We had picked midnight UTC without realising it was a choice.

Midnight UTC is not midnight

Midnight UTC arrives at very different times of day depending on where your visitors are:

The ranges are the two sides of daylight saving time.

For a European audience that moment is the middle of the night, so almost nobody is browsing and almost nobody gets split. For a US audience it lands in the early evening. Someone reads your pricing page at 7:50 pm, goes to dinner and comes back at 8:10 pm. The key changed at 8:00 (in winter it is 7:00). Same person, new identifier, two visitors.

That is why a tester in the US saw more visits than expected, and why a tester in Europe would never have noticed. We can’t give you a percentage, because it depends on how long your visitors stay, how many come back the same evening and when your traffic peaks. What we can say is what happened after the fix: the difference was big, and his funnels improved as well.

The funnels make sense once you see it. A funnel follows one visitor from step to step, so someone who looked at the pricing page at 7:50 pm and signed up at 8:10 pm was two visitors, and neither of them ever completed the funnel. Once the day was cut in the right place, his funnel numbers also matched what the insights in the dashboard showed.

Who this affects

Not everyone. It takes distance: if your visitors live close to the timezone where the day is cut, almost nobody is browsing when it turns over, and nobody gets split. It bites when visitors are far from that boundary. For a product like ours, where customers and visitors can be in every timezone, any single fixed boundary can land in somebody’s evening, so one boundary for everyone can’t be right.

A site whose visitors are mostly in one region can fix it by choosing that region’s timezone. A site with a truly worldwide audience can’t make it go away, because some part of the world is always in the evening when the day turns over. The best you can do is cut the day where your own traffic is lowest.

It wasn’t only visitors

When we looked, the visitor key was only one of two places that used the UTC day. The other was every daily bucket in the reports. We had two separate pieces of code that each decided on their own that a day starts at midnight UTC. They happened to agree with each other, so the mistake didn’t show up as a disagreement anywhere.

For a New York project, the bar labelled “Tuesday” actually held Monday 8 pm to Tuesday 8 pm. So it wasn’t just a few extra visitors. Every day in the report was shifted, and a newsletter sent at 9 pm on Monday would have appeared to work on Tuesday.

The fix: decide once, in one place

We gave every project a timezone and made it decide both things: the moment the key changes and the edges of every daily bucket. Now there is only one value, so the two can’t disagree.

It looked like a small change. It wasn’t.

• A counted day is frozen. We delete the raw rows of a day once we have counted it, for privacy. So when someone changes a project’s timezone, we can’t re-count old days on the new boundary, because the rows that would answer it are gone. A timezone change applies from then on, and the first day after a change is longer or shorter than 24 hours.
• Daylight saving makes days 23 or 25 hours long. An hourly chart has 25 buckets on the day the clocks go back and 23 when they go forward. We label each bucket with the UTC offset at that moment, so there is one bucket per real hour.

This is not only an analytics bug

Once we had seen it, we started spotting the same bug elsewhere. Plenty of software never decides whose day it is or when things reset, and just uses whatever clock the server has. Places we would look:

• Daily quotas and rate limits that reset at midnight UTC, so users elsewhere get their fresh allowance at an odd hour.
• Streaks and daily goals. A streak that breaks because the server’s midnight arrived mid-evening is a classic.
• “Today” in dashboards and reports, especially when you compare it with another system’s day, such as your payment provider’s.
• Daily budgets and billing periods that pace or reset by an account’s day.
• Scheduled digests and cron jobs. “Every day at 9 am” in whose 9 am?
• Anything that rotates an identifier on a schedule: session IDs, rate-limit buckets, A/B test bucket seeds. If it changes at a fixed time, it splits whoever is mid-action, exactly like ours did.
• Daylight saving, for everything that assumes a day has 24 hours.

What we would do differently

The first time DATE(timestamp) or “reset at midnight” shows up in code, ask whose midnight it is. We didn’t, and we found out when a tester in another timezone noticed.