Every busy service asks two questions all day long. First, “have we seen this item before?” Second, “how many different items have we seen?” Answering them exactly is easy for small data. However, once you reach millions of users, exact answers start to cost a lot of memory.
Fortunately, two clever structures solve this by accepting a small, predictable error. In this article, we look at Bloom filters and HyperLogLog, see where they hide in tools you already use, and measure how they behave with real numbers.
Why Approximate Answers Are Often Good Enough
Exact answers require remembering every item. Consequently, memory grows with the data, and lookups slow down as the data grows. Probabilistic structures break this link. They remember a compact summary instead of the items themselves, so their size stays small and fixed.
The trade is simple. You give up a little accuracy, and in return you save a lot of memory and time. For many tasks, such as skipping a pointless disk read or showing “about 1.2 million visitors”, that is a bargain.
Bloom Filters: “Definitely No” or “Probably Yes”
How it works
A Bloom filter is a long row of bits, all set to zero at the start. To add an item, you run it through a few hash functions. Each hash points to a position, and you flip those bits to one. To check an item, you hash it again and look at the same positions.
If any position is still zero, the item was never added. That answer is always right. If all positions are one, the item was probably added, but other items may have flipped those same bits by chance. Therefore, a Bloom filter can give false positives, yet it never gives false negatives. The idea dates back to Burton Bloom’s 1970 paper.
Where you meet it every day
Databases love Bloom filters. For example, Apache Cassandra keeps one per SSTable so it can skip files that cannot hold a requested row. According to the Cassandra documentation, typical false-positive settings range from 1% to 10%. A false positive only costs one needless disk read, so correctness is never at risk.
A tiny example you can run
Next, here is a minimal Bloom filter in plain Python. It needs no extra packages, so you can paste it into any terminal that has Python 3. It should print True and then False.
python3 - <<'PY'
import hashlib
class Bloom:
def __init__(self, size=1000, hashes=5):
self.size, self.hashes = size, hashes
self.bits = [False] * size
def _spots(self, item):
for i in range(self.hashes):
data = f"{i}:{item}".encode()
digest = hashlib.sha256(data).hexdigest()
yield int(digest, 16) % self.size
def add(self, item):
for s in self._spots(item):
self.bits[s] = True
def might_contain(self, item):
return all(self.bits[s] for s in self._spots(item))
bf = Bloom()
for user in ["ana", "nikos", "maria"]:
bf.add(user)
print(bf.might_contain("ana"))
print(bf.might_contain("zeus"))
PY
What the numbers say
To check the theory, I ran a larger test. I stored 10,000 items, then asked about 100,000 items that were never stored. The more bits we spend per item, the fewer false alarms we get.
| Bits per item | Hash functions | Measured false positives | Predicted |
|---|---|---|---|
| 4 | 3 | 14.85% | 14.69% |
| 6 | 4 | 5.58% | 5.61% |
| 8 | 6 | 2.13% | 2.16% |
| 10 | 7 | 0.81% | 0.82% |
| 12 | 8 | 0.32% | 0.31% |
Notably, measured and predicted values almost match. Moreover, going from 4 to 10 bits per item cuts mistakes from about 15% to under 1%, while ten bits is still just 1.25 bytes per item.
HyperLogLog: Counting Unique Things With Almost No Memory
The core idea
Imagine flipping a coin until you get heads. Getting five tails in a row is rare, so if someone tells you they managed it, they probably tried many times. HyperLogLog applies this logic to hashed items. It tracks the longest run of leading zeros it has seen, because longer runs suggest more distinct items.
One such guess is noisy. Therefore, HyperLogLog splits items across many small registers and combines them with a harmonic mean. The authors showed that the typical error is about 1.04 divided by the square root of the number of registers, as described in the original HyperLogLog paper.
Where you meet it every day
Redis ships HyperLogLog as a built-in type. As the Redis documentation explains, it uses at most 12 KB and gives a standard error of 0.81%, using the commands PFADD, PFCOUNT and PFMERGE. Because registers can be merged, you can count unique visitors per day and then combine seven days into one weekly figure without storing a single user ID.
What the numbers say
Again, I measured it. For each setting, I counted 100,000 unique IDs and averaged the error over 15 runs.
| Registers | Measured average error | Typical error (1.04 / √m) |
|---|---|---|
| 64 | 12.40% | 13.00% |
| 256 | 5.19% | 6.50% |
| 1,024 | 2.46% | 3.25% |
| 4,096 | 1.35% | 1.62% |
| 16,384 | 0.44% | 0.81% |
As a result, each fourfold increase in registers roughly halves the error. Keep in mind that an average absolute error normally lands a little below the standard error, and 15 runs is a small sample, so the last row is simply a lucky streak. Even so, the trend matches the formula well.
Bloom Filter vs HyperLogLog at a Glance
| Bloom filter | HyperLogLog | |
|---|---|---|
| Question answered | Is this item in the set? | How many unique items? |
| Type of error | False positives only | Small count error either way |
| Typical use | Skip needless database reads | Unique visitors, distinct users |
| Can list items? | No | No |
What We Have Learned
Probabilistic structures trade a small, controllable error for a big saving in memory. A Bloom filter answers “is it in the set?” and never misses an item that is really there, while its false-positive rate drops quickly as we add bits per item. HyperLogLog answers “how many unique items?” with an error of about 1.04 divided by the square root of its registers, which is why Redis can stay near 0.81% in 12 KB. Finally, our own measurements matched the theory closely, so you can tune both structures with confidence.
