# Inspect Parquet shards without building a full Arrow dataset cache

*By the RowCove team, developers of a commercial local data workspace. This
article is AI-assisted; the Python reference was executed and checked. All
example records are generated mock data. The reference works without RowCove.*

The question is small: which shard contains a suspicious training record, and what
does that record contain? The work can still be large. A normal dataset-loading
workflow can prepare another representation on disk before you inspect anything.
For a one-record investigation, that preparation may be more work than the task
needs. For repeated training epochs or indexed access, it may be worth doing.

A Hugging Face user reported this with roughly 500 GB of local Parquet shards:
[the loading operation created a large additional cache](https://discuss.huggingface.co/t/load-dataset-creates-a-duplicate-in-cache/175561).
That is one user's workflow report, not a RowCove customer or a measured result of
our own. It is a useful reminder to choose the operation before choosing the tool.

Hugging Face's [streaming documentation](https://huggingface.co/docs/datasets/en/stream)
describes local-file streaming without the complete Arrow conversion. It also
explains the tradeoff: an iterable dataset does not provide ordinary random row
access. A lookup near the end can still require traversing earlier records.
Streaming is useful; it does not make an arbitrary full-file question constant-time.

## Separate preview, search and preparation

Before starting, decide which answer you need:

- **A preview:** inspect schema and a few values, with a clear sample boundary.
- **An exact-ID search:** visit the required scope, retaining the source filename
  and file-local row ordinal. A negative answer is valid only when the scope is complete.
- **A prepared training dataset:** pay the conversion and storage cost deliberately
  when indexed access or the training pipeline requires it.

For a narrow Parquet investigation, select the columns needed for identification
before materializing Python objects. Read bounded batches and retain the source
location. Inspect the full nested payload of the selected record afterward.
For Arrow IPC, the source record-batch layout also matters: selecting fields from
an already obtained batch does not prove that physical reads or decompression
were reduced. Do not equate a small displayed table with a small I/O workload.


## Choose a narrower read before preparing the whole dataset

For a local Parquet shard, this is the small operation at the center of the
reference below:

```python
import pyarrow.parquet as pq

for batch in pq.ParquetFile("mock-000.parquet").iter_batches(
    batch_size=512, columns=["record_id", "status"]
):
    # Compare the ID in its original type; retain source and row location.
    # Continue through the required scope before claiming a negative result.
    pass
```

This projects Parquet columns, rather than creating a normal prepared Hugging
Face Dataset first. It does not build an index or promise a constant-time lookup.
A full scan is still a full scan, and duplicate IDs may produce multiple matches.
The standalone script below generates the named mock files before reading them.

## A reference with a known answer

The accompanying mock reference creates four Parquet shards and four Arrow IPC
shards, each containing 4,096 records. Its identifier exceeds JavaScript's safe
integer range, and each row includes a nested message payload. The inspection
step visits every record, compares typed IDs with PyArrow compute and materializes
only matching positions and scalar values as Python objects. It must find exactly one match in `mock-002`, at file-local
one-based row ordinal 18, and return the same answer for both formats.

The executed reference verifies 16,384 visited rows per format, at most 512 rows
per returned batch for this fixture, equal results, unchanged source hashes and
no extra files in the fixture directory. It does not invoke Hugging Face Datasets. The extra-file assertion covers only
the mock fixture directory; it is not a claim of zero OS, library or global cache.
It does **not** measure peak RAM, physical bytes read, OS cache activity, training
throughput or RowCove performance. Row count per batch is not a byte-memory limit,
and the IPC assertion depends on the fixture's explicitly chosen batch layout.

The reference then reopens the matched shard and checks the complete nested
message against its known value. Save the following as a Python file and run it
with Python 3.10+ and PyArrow installed (`python -m pip install pyarrow`).
Tested with Python 3.10.11 and PyArrow 23.0.1. Run `python inspect_records.py`.
It creates and removes its own temporary mock fixture. Hash verification reads
in chunks; it does not load an entire source file into a Python byte string.

```python
"""Mock correctness reference; no RowCove benchmark or cache-size comparison."""
import hashlib
import json
import tempfile
from pathlib import Path

import pyarrow as pa
import pyarrow.compute as pc
import pyarrow.parquet as pq


def digest(path):
    result = hashlib.sha256()
    with path.open('rb') as source:
        for block in iter(lambda: source.read(1024 * 1024), b''):
            result.update(block)
    return result.hexdigest()


def inspect(root):
    schema = pa.schema([
        ('record_id', pa.uint64()), ('status', pa.string()),
        ('messages', pa.list_(pa.struct([
            ('role', pa.string()), ('content', pa.string()),
        ]))),
    ])
    count, shard_rows, batch_rows = 4, 4096, 512
    first_id = 9007199254740993
    target = first_id + 2 * shard_rows + 17
    files = []
    for shard in range(count):
        rows = [
            {'record_id': first_id + shard * shard_rows + i,
             'status': 'review' if first_id + shard * shard_rows + i == target else 'ready',
             'messages': [{'role': 'user',
                           'content': f'Mock shard {shard} record {i}. ' + 'x' * 2048}]}
            for i in range(shard_rows)
        ]
        table = pa.Table.from_pylist(rows, schema=schema)
        parquet = root / f'mock-{shard:03d}.parquet'
        pq.write_table(table, parquet, row_group_size=1024)
        arrow = root / f'mock-{shard:03d}.arrow'
        with pa.OSFile(str(arrow), 'wb') as sink:
            with pa.ipc.new_file(sink, schema) as writer:
                writer.write_table(table, max_chunksize=batch_rows)
        files.extend([parquet, arrow])
    before = {p.name: digest(p) for p in files}
    expected = [('mock-002', 18, target, 'review')]
    results, observations = {}, {}
    for extension in ('parquet', 'arrow'):
        matches, seen, largest_batch = [], 0, 0
        for path in sorted(root.glob('*.' + extension)):
            file_row = 0

            def consume(batch):
                nonlocal file_row, largest_batch
                largest_batch = max(largest_batch, batch.num_rows)
                mask = pc.equal(
                    batch.column('record_id'), pa.scalar(target, type=pa.uint64()),
                )
                # Materialize match positions only; compare IDs as typed uint64.
                for position in pc.indices_nonzero(mask).to_pylist():
                    matches.append((path.stem, file_row + position + 1,
                                    batch.column('record_id')[position].as_py(),
                                    batch.column('status')[position].as_py()))
                file_row += batch.num_rows

            if extension == 'parquet':
                for batch in pq.ParquetFile(path).iter_batches(
                        batch_size=batch_rows, columns=['record_id', 'status']):
                    consume(batch)
            else:
                with pa.memory_map(str(path), 'r') as source:
                    reader = pa.ipc.open_file(source)
                    for index in range(reader.num_record_batches):
                        # Selection here does not prove physical I/O was pruned.
                        consume(reader.get_batch(index).select(['record_id', 'status']))
            seen += file_row
        assert seen == count * shard_rows
        # Fixture IPC batches were explicitly written at this size.
        assert largest_batch <= batch_rows
        assert matches == expected
        # Reopen only the matched shard and verify its complete nested value.
        ordinal = matches[0][1]
        path = root / (matches[0][0] + '.' + extension)
        def full_record(batches):
            remaining = ordinal - 1
            for batch in batches:
                if remaining < batch.num_rows:
                    return batch.slice(remaining, 1).to_pylist()[0]
                remaining -= batch.num_rows
            raise AssertionError('Matched source ordinal is missing')
        if extension == 'parquet':
            row = full_record(pq.ParquetFile(path).iter_batches(batch_size=batch_rows))
        else:
            with pa.memory_map(str(path), 'r') as source:
                reader = pa.ipc.open_file(source)
                row = full_record(
                    reader.get_batch(i) for i in range(reader.num_record_batches)
                )
        assert row == {
            'record_id': target, 'status': 'review',
            'messages': [{'role': 'user',
                          'content': 'Mock shard 2 record 17. ' + 'x' * 2048}],
        }
        results[extension] = matches
        observations[extension] = {'rowsVisited': seen, 'matches': len(matches),
                                   'largestReturnedBatchRows': largest_batch}
    assert results['parquet'] == results['arrow']
    assert before == {p.name: digest(p) for p in files}
    assert {p.name for p in root.iterdir()} == set(before)
    return {'mockOnly': True, 'formats': observations, 'exactId': target,
            'sourceHashesUnchanged': True, 'fullNestedTargetVerified': True,
            'noExtraFilesInFixtureDirectory': True,
            'huggingFaceDatasetsInvoked': False, 'performanceClaim': False,
            'peakMemoryMeasured': False, 'physicalReadBytesMeasured': False,
            'rowcoveAcceptance': False}


if __name__ == '__main__':
    with tempfile.TemporaryDirectory(prefix='rowcove-inspection-') as directory:
        print(json.dumps(inspect(Path(directory))))
```

## Evaluate the repeated investigation, including the expensive parts

A practical evaluation records opening/preparation, complete-query time, peak
memory, temporary storage and the time to verify an output. Count conversion or
index construction when the workflow requires it. Include failed and cancelled
operations; preserve the original shards.

RowCove is a commercial local data workspace for supported viewing, querying and
separate-copy editing on Windows and macOS. Its read-only collections preserve
source/partition context. Single-source Parquet/Arrow edit-copy work currently
has limits of 128 MiB, 200,000 rows and 512 columns; a collection is not a writable
training dataset. Preview shows up to 100 records read-only. Review the current
[reader capabilities](https://rowcove.com/formats) before planning a full workflow.

Authorized MCP can return bounded results to compatible AI clients. The local
workspace does not upload source files for ordinary local processing, but an
external AI client may transmit returned content to its provider. Keep sensitive
values out of demonstrations and review the tool scope before granting access.

If this is a one-off inspection, the reference above may be all you need. If
locating source shards, inspecting complete values and checking corrections is
part of your daily work, compare a dedicated workspace against your current
script-and-notebook workflow. Use your own known-answer question and count the
whole job, including preparation and output verification.

[See the local dataset workflow](https://rowcove.com/guides/query-dataset-shards?utm_source=hashnode)
or [try the read-only Preview on Windows or macOS](https://rowcove.com/download?utm_source=hashnode).
Collections remain read-only; Pro is required for supported full-file work, editing
and MCP. Check capabilities before choosing a plan.