How MarqIQ reads a very large database
Some analyses read every record you have. Others cannot, because the tool charges an API call per record or the records are too large to hold at once, and those read an even sample of your whole history rather than the newest slice, which would bias every trend. Whichever happened, the screen says so before the figures, and this article explains what to do with a figure that rests on a sample. If a single number looks wrong rather than a whole screen, read Two screens show different numbers for the same thing instead.
The rule, in one line
Every read is bounded, and every bound is told to you on the screen it applies to. Where a cap bites on your records, an even sample of the whole population is read in its place rather than the newest slice. A handful of reads are ordered rather than spread — the Email tab, Form fills, the Data Quality Center and write provenance — and the section below names each one and the order it takes.
How the reading works
Two things break when a workspace grows, and both are fixed the same way.
Pages follow an index, they do not skip
Asking a database for “rows 80,001 to 90,000” makes it sort the whole table again for every page. MarqIQ asks for “the next rows after this id” instead, on an index built for exactly that question. The cost of page 400 is the same as the cost of page one, so a four-million-record account does not time out where a forty-thousand-record one does not.
Above a cap, the sample is even
When a store is larger than an analysis’s cap, one record in every so many is read, chosen by a fixed rule applied to the record’s own id. Three things follow from that rule being fixed:
- The sample spans your whole history, from your oldest record to your newest.
- The same records are read on every visit, so two looks at unchanged data give the same figures. The number does not shift under you while you are reading it.
- Nothing about the sample can be chosen — not by you, not by MarqIQ — so it cannot be steered towards a flattering answer.
Why a sample is never the most recent records
This is the part worth understanding, because “the newest 80,000” sounds reasonable and is not.
Your newest leads have had the least time to buy. A sample of them contains almost no buyers, so a lead score judged on it looks worse than it is, and a funnel measured on it looks leakier than it is, because nobody in the sample has had time to reach the bottom of it. The same bias runs the other way on fields: a field your team started filling last spring looks fully filled, and one they stopped filling looks empty.
An even spread has neither problem. It contains the same mix of old and new records your database actually holds.
What the screen tells you
Every analysis prints one line about its population before any figure. It takes one of two shapes.
| When | What the line says |
|---|---|
| Everything in scope was read | “Read all 77,662 people.” Or, with a date range set, “Read 12,400 people created 1 Jul 2026 – 30 Sep 2026 (of 77,662 in all).” |
| A sample was read | “Read 6,000 of 41,208 people created 1 Jul 2026 – 30 Sep 2026 (of 77,662 in all): an even sample spread across the whole period, because field history is one call per record. Shares are reliable; counts are the sample’s.” |
Shares are reliable; counts are the sample’s
That last sentence is the one to carry into a board pack. On a sampled read, “31% of deals have no source” describes your database. “1,482 deals have no source” describes the sample, and the real figure is proportionally larger. Quote the share, or quote the count with the sample size attached.
What each analysis reads, and up to how much
The cap differs by analysis because the constraint differs. Reading a field’s change history costs one API call per record; counting how many people carry a value is one database query however many there are.
| Analysis | How much is read | Why that bound |
|---|---|---|
| Source truth | Every person in scope, with no cap. The one exception is the card that compares what your tool recorded against what MarqIQ captured, which matches on the first 20,000 form fills MarqIQ holds | Counting values is a database aggregate and costs the same at any size. The comparison joins those counts to form fills by email address, which does not |
| Audience health | Every person in scope, with no cap | The same — aggregates over the whole table |
| Field discipline | Up to 600 people and 600 deals per tool on HubSpot, Marketo and Salesforce; up to 60 each on Pipedrive, Zoho CRM and Dynamics 365. Up to 30 fields per object | Change history is an API call per record on the second group, and a batched call on the first. The screen names the tool and the number |
| Forecast accuracy | Up to 2,000 closed deals | HubSpot serves deal history 100 at a time and Salesforce answers a history query per chunk, so 2,000 is about twenty calls |
| Lead scoring | Up to 80,000 people, in chunks of 10,000. Score fields are discovered from a 3,000-record sample, up to 60 fields | Each record carries its whole property bag, so memory is the bound rather than calls |
| Funnel | Up to 80,000 people, in chunks of 10,000 | The same |
| Form fills | Up to 60,000 form submissions | This one is newest-first rather than an even spread, and the screen says when it was capped — see the caveat below |
| Per tool: the 2,000 most recently published HubSpot emails; the 5,000 most recent Marketing Cloud sends kept from up to 25,000 read; the first 5,000 campaign reports Mailchimp lists for the period; the first 45,000 Marketo email activities, or 40 seconds of walking its stream, whichever comes first | Each tool serves its results a page at a time, and MarqIQ stops after a fixed number of pages. Where a bound bites, the note on the card says how much was covered — see the section below | |
| Automations | Every automation the tool lists | Nothing is sampled. Per-tool limits are stated on that card |
| Screen | How much is read | Why that bound |
|---|---|---|
| Field Intelligence | 6,000 people, 20,000 deals or 10,000 companies per object, as an even spread above those numbers | Contacts carry several kilobytes of properties each |
| Data Audit and Executive Read | Every deal in scope. Touch paths read up to 20,000 linked contacts | Deals are few and small; the contact join is the expensive part |
| Integration Audit Report field completeness | Every synced contact, in batches of 1,000, with a 40-second budget | Nothing is sampled. Where the budget runs out it says “covers the first N of M records” |
| Integration Audit Report data quality | The whole contact table, as exact counts | Duplicate and missing-email counts are database aggregates |
| Data Quality Center | Up to 60,000 profiles, email-holders first, newest-created first | Identity resolution holds every record in memory at once. This one is newest-first — see the caveat below |
| The records behind a finding | Up to 200 record ids per finding, spread evenly across the set found | A list of tens of thousands of ids in a stored document is not a list anyone reads |
The reads that are not an even spread, said plainly
These reads take an order rather than an even spread, and each says so where it is used.
- Form fills reads the 60,000 most recent submissions. A form launched last month is fully represented; a form retired three years ago may be absent from a very large store. Narrow the date range to bring an older period fully into view.
- The Data Quality Center scores the newest 60,000 profiles, with records carrying an email filling the window first. On a store larger than that, the score describes your recent records rather than your whole history.
- Write provenance on the Field Intelligence tab reads the 100 newest records on purpose, because the question it answers is “what writes this field today”. It runs on HubSpot alone, because no other tool reports what wrote each value.
- The Email tab reads the 2,000 most recently published HubSpot emails. On Marketing Cloud it reads up to 25,000 sends and keeps the 5,000 most recent, and every figure rests on what it kept. The note on the card says so in as many words once the kept history is full: it names the number your rates are built on, and says that older sends are in no figure on the tab.
- Two email tools are bounded without being newest-first. Mailchimp gives the first 5,000 campaign reports it lists for the period, in its own order. Marketo is walked forward from the start of the period and stops at 45,000 email activities or 40 seconds, whichever comes first, so on a long period it is the earliest stretch that is covered and the latest that is missing. The card says how many activities it read and asks you to pick a shorter period.
- The Marketo bound counts activities, not sends. Opens and clicks are activities too, so a period with heavy engagement reaches 45,000 after fewer emails than a quiet one.
What this does to a finding’s percentage
A share measured on an even sample is an estimate of the share in the whole population, and it is a good one — but it is an estimate, and it is worth knowing how MarqIQ handles that.
- Where a Field discipline finding can, it prints the conservative figure as well: “On 84 of 600 deals checked (14%). Across all 31,900 deals that points to at least 11%.” The second number is the floor, not the estimate.
- A field is judged at all only where change history came back on at least 20 records. Below that the field is listed as untested rather than sound.
- A finding then needs a minimum before it fires — at least three affected records and at least 2% of those checked — so one odd record in a 600-record sample never becomes a finding.
- A count on a sampled screen is the sample’s count. Multiply nothing yourself; read the share, which is the figure that carries across.
- A screen that reads everything prints no sample sentence, and its counts are the real counts.
How to widen the picture
You can usually turn a sampled read into a complete one, because every cap applies to the records in scope rather than to your whole store.
Set a Created range at the top of the tool’s page.
The range filters by the tool’s own created date, in the query, before any cap applies.
Narrow it until the in-range count falls under the cap.
The population line tells you both numbers, so you can see when the sample sentence disappears.
Read the same analysis a period at a time.
Four quarters read whole are four complete readings. They are more work than one sampled reading of the year, and they are exact.
On Dynamics 365, switch on auditing for the columns you care about first.
Dynamics keeps a column’s history only where auditing is on for the organisation, the table and that column. The Field discipline screen names the fields that are not audited and where to switch it on; history starts from that day, not retrospectively.
Tip
A ranged view lives in the address bar, so a colleague opening your link sees the same population and the same figures.
What is never done
- No figure is estimated upwards from a sample and then printed as a count.
- No cap that shapes a figure is applied without being stated on the screen where that figure appears. There are no footnotes.
- No sample is drawn from a subset you cannot see. Where a read is spread it is the whole population in scope or an even spread of it; where a read is ordered — the Email tab, Form fills, the Data Quality Center, write provenance — the row above names the end it takes.
- Records you have deleted in the tool are excluded everywhere, before any count.
- On Pipedrive, anything in the Leads Inbox is set aside before deals are counted, and the count set aside is stated.
If it did not work
Two analyses on the same tool report different record counts
Why: They have different caps, and a sampled read counts its sample. Source truth reads every person; Field discipline reads up to 600.
Fix: Compare the population lines rather than the counts. Both state what they read, out of what.
A count fell after you narrowed the date range, but the share went up
Why: The range changed the population, so both the numerator and the denominator changed. That is the range working, not a fault.
Fix: Read the population line for each view. A share is only comparable against the same range.
A sampled figure moved between two visits although nothing was synced
Why: It should not. The sample is fixed by record id, and the stored result is reused until the underlying records change.
Fix: Check the last-sync date under the page heading. Where it has moved, records changed; where it has not, tell support — a figure moving on unchanged data is a defect.
“Field completeness covers the first N of M records”
Why: The Integration Audit Report scans every contact rather than sampling, and this one has a 40-second budget. It stops and says how far it got.
Fix: Read those fill rates as describing N records. The duplicate and missing-email counts on the same page are whole-table either way.
A Field discipline finding you expected did not appear on a very large store
Why: It needs at least three affected records and at least 2% of those checked. In a 600-record sample of four million, a problem on a small share of records can fall under the floor.
Fix: Narrow the range to the period you suspect, so more of that period lands in the 600, and run it again.
Next steps
Related articles
Was this article helpful?