What Is a Data Quality Check?

Scrapeless Web Unlocker retrieves web content that your application can inspect and validate before accepting extracted records into a dataset.

What Is a Data Quality Check?

A data quality check is an explicit test of whether data meets a requirement for its intended use. The requirement might concern a field, a relationship between records, or the coverage of an entire dataset. A check produces evidence: which rule ran, which input it evaluated, what it observed, and whether the result met the acceptance condition.

For a scraped catalog, a field check might require a product identifier. A relationship check might require each observation to reference a known product. A dataset check might require every approved category to have a recorded collection outcome. Passing the first check says nothing about the other two. Quality is a set of decisions tied to use, rather than a single universal score.

Which Quality Dimensions Can You Test?

Quality dimensions describe different ways data can fail its purpose. Completeness asks whether required information is present. Validity asks whether a value follows an agreed structure or rule. Consistency asks whether related representations agree. Uniqueness asks whether the intended key occurs as expected. Timeliness asks whether data is available within a useful period. Accuracy asks whether the value corresponds to the thing it claims to describe.

These dimensions overlap, but they are not interchangeable. A complete list of incorrect addresses is not accurate. A current observation can contain an invalid currency code. A unique product key can point to an unrelated image. Name the dimension so the person responding knows what evidence the check provides.

The Data Quality Vocabulary distinguishes a quality dimension from a metric that measures it. In operational terms, completeness is the concern; the proportion of required identifiers present in a defined batch is a metric. The threshold and the business response remain decisions for the dataset's owner.

How Do You Turn a Requirement Into a Check?

A useful check specifies its scope, calculation, acceptance condition, and consequence. Start with a plain requirement such as: every accepted price observation must identify its product and currency. Then define which rows are eligible, how missing values are represented, and which fields form the observation key.

Scope controls the denominator. If a field is required only for items offered for sale, withdrawn listings should not automatically count as failures. Conversely, removing every difficult row from the denominator can make the metric look healthy while reducing the dataset's usefulness. Keep the eligibility rule visible alongside the result.

Thresholds need a reason. A required primary key may justify rejection of any affected record. A small change in optional image coverage may justify an investigation rather than stopping an entire load. Separate hard contract violations from anomaly signals, and document who can approve an exception.

A result should identify the rule version and the input boundary. When a threshold changes, historical results should remain interpretable. Otherwise an apparent improvement may reflect a looser rule rather than better data.

Where Should Checks Run in a Scraping Workflow?

Place checks where they can prevent an incorrect interpretation from spreading. At collection, verify the intended page or response is present. At extraction, verify candidate records contain expected identifiers and content. At normalization, verify types, units, and enumerated values. Before delivery, verify keys, relationships, and batch coverage.

Scrapeless Web Unlocker supports the page-access stage; it does not replace your business acceptance rules. Its web content collection capabilities help retrieve the material to inspect. Your application must still distinguish a catalog from a consent screen, a source record from a navigation card, and an absent value from a parser failure.

For text datasets, clean web text for retrieval pipelines illustrates why successful page retrieval is only the beginning. Menus and access messages are valid text. They become bad training or retrieval material when treated as the intended article.

What Does an Illustrative Quality Review Look Like?

A quality review connects observed failures to a concrete consumer risk. Consider an illustrative price-comparison dataset, not a measured service result. Each record contains source product identity, amount, currency, availability, and observation context. The consumer compares offers for the same item within the same market.

The first review asks whether identity is stable across collection runs. The next checks whether amount is numeric and currency is explicit. A cross-field rule marks a sale amount greater than the regular amount for inspection if that contradicts the agreed source interpretation. A coverage check reconciles approved categories against recorded traversal outcomes.

Suppose a category yields no records. Treat that as an unresolved outcome until the evidence distinguishes an empty category from a blocked response or changed selector. Do not populate a healthy zero merely because the extractor returned an empty list. A zero count is a claim about the source; an extraction failure is a claim about the workflow.

Finally, inspect selected records against retained evidence where retention is permitted. Automated constraints can detect impossible shapes, but they cannot alone determine whether the amount belongs to the main product, a suggested item, or a financing installment. Record that interpretive distinction in the parser contract.

How Do Schema Checks Differ From Business Checks?

Schema checks evaluate allowed structure and types, while business checks evaluate meaning within the use case. A record may satisfy a schema requiring a string product key and a numeric amount, yet combine fields from different products. That record passes structural validation and fails business interpretation.

Key and relationship constraints provide another layer. The tabular data model's validation rules describe errors such as duplicate primary keys and unresolved foreign-key references. These rules are valuable when extracted rows are assembled into linked tables, but they depend on choosing the right grain and reference data.

For graph datasets, SHACL shape validation can express constraints on nodes and their properties. A conforming graph satisfies the tested shapes; conformance is not proof that every statement is true. The same boundary applies to tabular validators and API schemas.

How Should Failures Be Handled and Reported?

A failed check should lead to a recorded decision that protects the intended consumer. Isolate affected records when other records remain valid and the consumer can tolerate partial coverage. Hold the whole partition when missing coverage would make comparisons misleading. Preserve the reason so downstream systems do not interpret an exclusion as source absence.

Quarantine stores require their own controls. Rejected content may contain unnecessary personal information or secrets accidentally captured from an input. Retain only what investigation needs, limit access, and set a deletion policy. Quality diagnostics should not become an unrestricted copy of every source response.

Reports should show evaluated records, affected records, rule identity, source scope, and disposition. A percentage without a denominator hides scale. A green dashboard without a coverage statement hides missing inputs. Keep accepted, excluded, unresolved, and unavailable outcomes distinct.

Track changes over time by source and rule rather than relying only on a global average. An aggregate can hide a broken category when another category grows. Assign an owner and a review path before increasing collection volume.

How Do You Keep a Check Suite Useful?

A check suite remains useful when its rules follow real consumer requirements and its signals stay explainable. Remove redundant alerts, revise assumptions when source behavior changes, and preserve meaningful failures as regression cases using approved, minimized examples. Test both the rule and its disposition path.

Collection cost is only part of the operating picture. Scrapeless pricing helps estimate access costs, while review time and rejected output affect cost per accepted record. Strong acceptance rules make that comparison possible because the denominator represents useful data.

Conclusion

Begin with the errors that would change a consumer's decision. Express each as a scoped rule, capture evidence, and decide what a failure means before the load runs. Expand the suite when a new failure mode justifies a new check.