Grid Data Archive

The GDA data lake — read-only.

Direct access to the parquet data lake behind the Twin Scroll Grid Balancer evidence base — over half a trillion data points in the full system across 782 datasets — this public lake exposes ~144 billion of them — and growing: 350 billion+ from official GB grid feeds, 150 billion+ derived and inferred, plus a full power-flow solve of the grid for every second of the last 25 years (800 million+ solves). New data is captured continuously, so it only gets bigger. You mount it as a folder and read the parquet. Absolutely read-only: you can browse and download, nothing more.

The lake is served over SFTP from a locked-down account. Download the key, point sshfs or sftp at it, and you have the whole parquet tree as a normal read-only filesystem — query it with DuckDB, pandas, polars, or anything that reads parquet.

500bn+

data points · full system

782

datasets

200+ GB

parquet

~945k

parquet files

25 yr

solved every second

Live 500,000,000,000 data points across the full system, and counting (~144 billion in this public lake).

A living dataset — new grid data is captured continuously; the total above extrapolates from a measured anchor and is re-trued periodically.

1 Get the key

Access is by SSH key only. Download it, then keep it somewhere sensible.

mountservice_ed25519

Ed25519 private key · read-only mountservice account

Key fingerprint: SHA256:eKvQNEXJs0EVqhtPAVia6o7GSmHFo5wTtfpqxk2Gx78

Download key

On macOS / Linux, lock the permissions before using it:

chmod 600 ~/Downloads/mountservice_ed25519

2 Connection details

Host	ingest.twinscrollgridbalancer.co.uk
Port	2222
User	mountservice
Protocol	SFTP (SSH)
Path	/lake
Auth	the key you just downloaded

3 Mount the lake as a folder

The easiest way — sshfs mounts the whole lake as a local read-only directory, so every tool just sees files on disk.

macOS / Linux

mkdir -p ~/gda-lake sshfs [email protected]:/lake ~/gda-lake \ -p 2222 -o ro,reconnect,IdentityFile=~/Downloads/mountservice_ed25519 # browse it like any folder ls ~/gda-lake # unmount when done umount ~/gda-lake

Needs sshfs — macOS: macFUSE + sshfs; Debian/Ubuntu: sudo apt install sshfs. On Windows use WinFsp + SSHFS-Win, or just use WinSCP (below).

Prefer to browse & pull single files? Use sftp

sftp -P 2222 -i ~/Downloads/mountservice_ed25519 [email protected] # then, at the sftp> prompt: cd /lake ls get DataSources/GridWatch/... /local/path/

On Windows, WinSCP (New Site → SFTP → host ingest.twinscrollgridbalancer.co.uk, port 2222, user mountservice, and set the private key file) gives a drag-and-drop window.

First connection shows the server host-key fingerprint — accepting it is normal.

4 What's in the lake

Over half a trillion data points across 782 datasets in the full system (and growing) — ~144 billion of them in this public lake, chiefly under /lake/DataSources — in three top-level trees:

/lake/DataSources

350 billion+ data points. Raw official source feeds — NESO / National Grid, the DNOs (UK Power Networks, SSEN…), GridWatch, Elexon, OSM network geometry, weather and more. The bulk of the lake.

/lake/InferedOrDerivedData

150 billion+ data points. Datasets derived or inferred from the raw feeds — cleaned, temporally aligned, gap-filled and enriched. The evidence base most of the analysis runs on.

/lake/GDA-SOLVES

The solved grid-state library, reconstructed purely from public data, at two fidelities: a reduced 10-zone model solved for every second of every day, 25 years deep (800 million+ solves, 100.0000% converged, in full-history/), and a full 2,227-bus UK network solve (full-uk/) — ~3,000 daily full-network snapshots sweeping the half-hourly periods, 6.7 million bus-level results. The per-second grid is also downloadable a day at a time from the Solves page.

Everything is parquet, partitioned into dated folders. Only data is published here — none of the ingestion, processing or solve code.

5 Read the parquet

Once mounted, point any parquet-aware tool at the files. A few starting points:

DuckDB (fastest for ad-hoc SQL over many files)

-- glob straight over the mounted tree SELECT * FROM read_parquet('~/gda-lake/DataSources/GridWatch/**/*.parquet') LIMIT 20;

Python — pandas

import pandas as pd df = pd.read_parquet('~/gda-lake/GDA-SOLVES/_index/idx_20260621.parquet') print(df.head())

Python — polars (lazy, out-of-core)

import polars as pl lf = pl.scan_parquet('~/gda-lake/DataSources/**/*.parquet') print(lf.select(pl.len()).collect())

6 The ground rules

Read-only, enforced. The account cannot write, rename or delete anything — the mount is read-only, the SFTP server is read-only, and the account is confined to /lake. Nothing you do can affect the lake.

Look, don't copy. This data is provided for access and analysis — not for redistribution. Some raw feeds (NESO / DNO) carry their own terms. Access is metered: normal reads and random-access querying are fine, but attempting to bulk-download a copy of the corpus trips a per-source data limit and your access is automatically cut off. Don't mirror the lake.

Please be gentle — a couple of parallel connections, paced reads. Need a larger or bulk allowance for a legitimate use? [email protected].