GoBase: A Queryable Catalog for Data Lakes
A catalog and retrieval layer that indexes an entire data lake, raster, vector, documents, spreadsheets, images and video, into one queryable spine with a natural-language layer on top.
Data Infrastructure · Geospatial Catalog & Retrieval
The problem
Most organisations that work with geospatial data have a lake rather than a catalog. Files are scattered across formats and folders, identified by filename rather than by content, so the same raster arrives three times under three names and nobody can tell which copy is current. There is no single index across modalities, which means a question that spans a raster, a report and a vector layer has no place to be asked. And there is no plain-language entry point: finding anything assumes you already know where it is and what it is called.
The approach
GoBase is a catalog and retrieval layer that indexes the whole lake into one queryable spine: raster, vector, documents, spreadsheets, images and video. Ingestion is content-addressed and idempotent, so re-running it over the same lake does not duplicate anything. On top of the catalog sits a natural-language query layer, and underneath it a two-level retrieval scheme that first finds the right document or asset and then the passage inside it. It runs offline and on-premise, which is often the condition for pointing it at a client’s real data in the first place.
The content-hash spine
Every asset is keyed by the hash of its own bytes rather than by its path. Two copies of the same file under different names collapse to one entry, a changed file becomes a new version of a tracked asset rather than an unrelated row, and ingestion can be re-run safely because the identity of an asset never depends on where it happened to sit. This is the part that makes a lake countable: you can finally say how many distinct things are in it.
A detail table per modality
A single schema cannot describe a shapefile, a spreadsheet and a drone video equally well, so the spine carries a detail table per modality: vector, raster, docs and tabular, each recording what is actually interesting about that kind of file. There is also an explicit unparsed catch-all, so a file the parsers do not understand is still catalogued as an asset instead of vanishing. Coverage of the lake stays complete even where understanding of it is partial.
Indexing: spatial, topical and semantic
Three indexes run over the catalog in parallel. PostGIS holds each asset’s spatial extent, so proximity and intersection are first-class query terms. An ltree topic hierarchy holds where an asset sits thematically. And pgvector holds embeddings for semantic matching, so a question can land on relevant material that shares no keywords with it. A query can therefore be constrained by place, by subject and by meaning at once.
Two-level retrieval
Retrieval happens twice. The first pass ranks across the catalog at the level of a document card, deciding which assets are worth opening at all. The second pass runs chunk-level retrieval inside the shortlist to find the specific passage or region that answers the question. Splitting it this way keeps recall high across a large lake without paying to embed and rerank every chunk of every file for every question.
Cited answers
The output is a short, ranked set of results with their provenance attached. A question such as "which reports mention flood risk near the port?" comes back with the raster whose extent intersects the port, the document and the page inside it with the cited passage and its bounding box, and the vector layer two kilometres away, each labelled with why it matched. Answers point at the client’s own files, so a reader can check the source rather than trust the summary.
The handoff to GoPilot
GoBase is a product on its own, and it is also the knowledge side of GoSuite. GoPilot queries it as its retrieval-augmented generation layer, which is what lets the agent answer from a client’s own archive instead of guessing, and what lets it cite the archive when it does. The catalog supplies the evidence; the agent supplies the workflow.
The result
A heterogeneous lake becomes one searchable catalog with cited answers. Ingestion is idempotent, so the catalog can be kept current by re-running it rather than by curating it, and the retrieval layer can answer questions over data that was previously unreachable in practice: present on disk, but not findable by anyone who did not already know it was there.