Data lake

Huntbase can query telemetry that lives in a data lake as well as data it reaches through a connection. There are two ways in, and you can use both at once:

• Bring your own lake — point Huntbase at an Apache Iceberg store you already own. Your raw events stay in your bucket and Huntbase queries them in place. Your lake is read-only to Huntbase: it never sends events to it.
• Ship logs to Huntbase — send events from a log shipper (Fluent Bit, Vector, rsyslog, syslog-ng) straight into the Huntbase data lake. This needs no store of your own, and is the shortest way to get data in.

Either way you query the result with KQL (Huntbase Lake) or Search (Huntbase Lake) in the Explorer. One query can read the Huntbase data lake and your own lakes together, but reading your lake costs you scans of your bucket, so Huntbase reads it only when you ask. See Querying the lake.

Beta

The data lake is currently rolling out and may not be enabled for your organization.

Bring your own lake

A lake store is set up like any other connection — see Create a connection — with one difference that matters, covered below.

Huntbase mounts your catalog read-only and queries the tables where they are — this is called federation. Nothing is copied into Huntbase, and nothing is written back to your lake: ingest keys can't send to it, and detections found in your data are stored in Huntbase, not in your bucket.

Choose how your tables are catalogued

The first question on the Connect step is How your tables are catalogued, and it decides what else you are asked for. There are three answers:

Catalog type	Use it when	What Huntbase then asks for
Give us a bucket — Huntbase runs the catalog	You have object storage and nothing else. This is the default and the shortest path.	Bucket / warehouse location
AWS Glue Data Catalog	You are on AWS and already use Glue. Nothing to stand up.	Bucket / warehouse location, and a Glue Catalog ID only if the catalog is in a different AWS account from your key
I already run an Iceberg REST catalog	You operate an Iceberg REST catalog service of your own.	The catalog's URI

The form only shows the fields your answer needs, so switching the catalog type changes which settings appear.

What to have ready

You'll need	Notes
Object store endpoint	Pick your AWS regional endpoint from the list, or type your own for MinIO, Ceph, Cloudflare R2 or any other S3-compatible store.
Region	Pick from the AWS regions, or type the region string your store expects. us-east-1 is the usual answer for MinIO.
Access key and secret key	S3-compatible credentials. Read access to the warehouse prefix is enough for queries; write is used only for Iceberg table metadata.
Catalog settings	Whichever the table above says your catalog type needs.

Apache Iceberg only

Bring your own lake reads Apache Iceberg tables over Parquet in an S3-compatible store. Delta Lake tables are not read directly.

The rest catalog type currently connects to an Iceberg REST catalog that does not require authentication. Managed catalogs that speak the same protocol but authenticate — Databricks Unity Catalog, Snowflake Open Catalog, Apache Polaris, AWS S3 Tables — are not connectable yet. Support for them is planned; talk to us if you need one.

Creating and provisioning the store

Search for the lake product on the Product step, fill in the settings and keys on Connect, then click Create & check.

Creating the connection is only half of it. Unlike every other product, a lake store has to be provisioned before it will answer anything, and that starts automatically when the connection is created. The Verify step shows the progress:

Step	What is happening
Reach the lake	Connecting to the Iceberg catalog and object store.
Register catalog	Creating the store's query catalog.
Attach tables	Binding the telemetry tables Huntbase reads.
Join federation	Including the store in federated lake queries.

Provisioning continues if you close the page. You can follow it, and re-run it, from the Lake store card on the connection's own page.

warning

Until provisioning completes the store answers nothing, and lake queries skip it. A store that is merely healthy is not necessarily queryable — check the federation badge below.

In federation, or excluded

The Lake store card carries a badge that answers the only question that matters for results:

Badge	Meaning
In federation	The store is active and provisioned, so lake queries read it.
Excluded	Lake queries skip it. Hover the badge for the reason — never provisioned, provisioning failed, provisioning still running, or the connection is inactive.

This matters because an excluded store makes your results narrower without making them look wrong. When a query skips a store, the results panel says so — see Which stores answered.

Telemetry › Destinations shows every lake of yours side by side under Your lakes — Huntbase reads these: whether it's in federation, and its readiness trail. Where the Telemetry page is available, a lake's connection page links to Browse with your lakes included.

When provisioning fails

The trail names the step that failed and gives you a short remediation. The technical diagnostic is deliberately terse (something like probe: ConnectionRefusedError) because Huntbase does not echo back what your catalog endpoint returned. Treat it as a hint, and check in this order:

1. Are the object store endpoint — and the catalog URI, if you supplied one — reachable from outside your network?
2. Do the access keys still work, and does their policy cover the warehouse prefix?
3. Are the region and warehouse location right?

Then use Retry provisioning.

Ship logs to Huntbase

If you would rather send events than expose a store, mint a telemetry ingest key and point a log shipper at Huntbase. Every key sends to the Huntbase data lake. Keys are never bound to a lake of your own, which Huntbase only reads.

Mint a key

Click New ingest key on the Telemetry page. Onboarding's Telemetry step mints keys too. Where the Telemetry page isn't available, use the Send telemetry to Huntbase card on the Connections page, or Telemetry ingest keys under it, and click New key.

If you see Request access instead of your sources, ingest is not enabled for the organization you are working in. Use the button to ask — it fills in the details support needs to turn it on.

Give it a Source label that names the sender — something like edge-fluentbit-eu. One key per shipper or source means a single noisy or compromised sender can be revoked on its own.

warning

The token is shown once, when you create it. It is stored only as a hash, so it cannot be recovered or shown again. Copy it before you close the dialog — the dialog will ask you to confirm if you try to dismiss it without doing so.

Afterwards the key is identified only by its prefix (for example hbik_3f9a2c1).

Where to send it

Ingest endpoint under the table on Telemetry › Sources (or the Send telemetry to Huntbase card) shows the endpoint for your deployment, so what you read there is what you paste. It is the same host this app talks to:

Method and path	POST /api/v1/telemetry/ingest
Host	Your Huntbase host, shown on the card.
Port	443
Auth	Authorization: Bearer <your ingest key>

Shippers that want host, port and path separately (Fluent Bit, for example) will find them listed individually there.

Before wiring anything up, you can prove the network path from the machine that will do the shipping:

curl -fsS https://<your-huntbase-host>/api/v1/telemetry/ingest/health

That probe is unauthenticated by design, so it works before any key exists — and it distinguishes "the firewall blocks this" from "the key is wrong", which is otherwise a long afternoon.

Configure your shipper

The mint dialog generates ready-to-paste configuration for Fluent Bit, Vector, rsyslog and syslog-ng, plus a command that checks Huntbase is reachable before you involve any secret. Its curl tab sends one test event with the new key, so you can see it arrive before you configure a shipper; the endpoint answers {"accepted":1,"rejected":0}. You can bring the configuration back up later with ⋯ › Shipper config on Telemetry › Sources (or the key's Config action where the Telemetry page isn't available) — with a placeholder in place of the token, since we cannot show it again.

What the endpoint accepts:

• A single JSON object, a JSON array, or newline-delimited JSON.
• Optionally gzipped.
• Up to 10 MB per request, before and after decompression.

It answers with a count of accepted and rejected records. A malformed record is counted, not retried, and never fails the rest of the batch — so a shipper should treat the response as success and not resend.

What Huntbase adds to each event

Nothing changes in what you send, and the event is stored exactly as it arrived. Alongside it, Huntbase records a few things it works out when it accepts the event:

Added	How
Event time	The event's own timestamp, from the first of time, timestamp, @timestamp, ts or eventTime that can be read as a time (ISO 8601, or epoch seconds, milliseconds, microseconds or nanoseconds), in UTC.
Host	The first of host (a string, or an object with name or hostname), hostname, host.name or computer.
Source label	The label of the ingest key the event arrived on.
Shipper	Fluent Bit, Vector, rsyslog, syslog-ng or other, recognized from the shipper's User-Agent.

These values give every event the time, host and source fields that Search telemetry filters and charts on. They also give the Telemetry Overview its ingest lag. To get the most from them, send a timestamp and a host name in one of those fields. You don't have to rename anything you already send.

Revoking a key

On Telemetry › Sources, ⋯ › Revoke, then confirm. Shippers using the key start failing immediately. The row stays listed, marked Revoked, so the history of what existed is preserved.

Status and Last event are the quickest way to tell a key that is wired up and working from one that was minted and never configured. Status is based on the last accepted event, so a key that is used but only sends records that are refused doesn't show as receiving. See Source statuses.

Querying the lake

Pick KQL (Huntbase Lake) in the query bar's language picker. Unlike the other languages it needs no connection, because a lake query can search Huntbase's own store and every one of your stores that is in federation together. If you have no lakes of your own, the bar shows Data lake instead of a connection picker.

When you do have lakes in federation, a Stores pill takes its place. It shows what Run reads: by default the Huntbase data lake and every one of your lakes in federation (Lake + N). Pick Huntbase Lake only to leave your lakes out. The line under a new tab's input names the stores a run will read, and the compact bar's summary names them after a run.

When your lake is read

Reading your lake scans your bucket, which you pay for, so nothing reads it on a guess:

Where	Reads your lake?
KQL (Huntbase Lake) run from Query	Yes, every store in federation, unless you pick Huntbase Lake only in the Stores pill.
A KQL (Huntbase Lake) query that Auto runs when you press Enter	No — the Huntbase data lake only.
Search (Huntbase Lake)	Only when you ask: press ⌘/Ctrl+Enter instead of Enter, or include your lakes in the stores pill. See Search telemetry.
Scout	Yes, within your scope and permissions.
Check against history on a watcher	Yes, every store in federation.
Live tail	No — the Huntbase data lake only.
A watcher's readiness chip	No — it checks the Huntbase data lake only.
The entity timeline	Only when you click Include it in its note.

Daily scan budget

Before Huntbase reads your lake it estimates how much of your bucket the read will scan, and charges the estimate against your organization's Daily scan budget for your own lake (50 GB a day unless an admin changes it — see Capabilities). The budget is shared by every reader, and the Huntbase data lake never counts against it.

When a query would go over what's left of today's budget, your lakes drop out of that run and the Huntbase data lake part still runs. The results say so: your stores are listed as excluded because the scan budget is used up. A search that would go over asks you to narrow the view instead.

Which stores answered

Because one query can read several stores, the results panel tells you what it actually searched:

Notice	What it means
Searched N stores	Expand it to see which. Each store can be used to narrow the results to just its rows.
N stores were excluded	A warning, not a footnote: the store was unavailable, isn't answering, or today's scan budget is used up, so your results may be incomplete. The notice links to the connection.
Your lake wasn't searched	The run didn't ask for your lake — for example a query that Auto ran. Click Include it to run it again with your lake.
Showing the first N rows	A row cap was applied. Add `
A column is empty for some events	No mapping exists for that field in those events, so those rows are empty rather than guessed.

A lake that stops answering doesn't fail the whole query. Huntbase checks it, leaves it out of the run, and the notice says it isn't answering, so results cover your other lakes. It's left out of other reads for a few minutes, then tried again.

Events in your lake written as plain OCSF 1.x fill the same hb_ surface columns as the formats Huntbase maps itself, so one surface query covers both. In Search, each OCSF event's summary line shows the fields that matter for its class: for a sign-in, for example, the outcome, user and source address, with the field your search matched highlighted.

Every result row carries a store column showing which store it came from, and you can narrow to one store from there.

To look at shipped events without writing a query, use Search telemetry. Search filters by clicking and shows the same KQL (Huntbase Lake) query it runs.

Next steps

• Query languages — writing lake KQL, and what the dialect supports
• Connections — creating and managing connections
• Telemetry — the health of each telemetry source and its ingest key
• Search telemetry — filter shipped events by clicking
• Watchers — standing rules over the data you have connected