Morning HN - - I have a quick story about building a startup and misreading the market, and an ask. (The ask: Try out our product>> https://app.datable.io/auth/sandbox)
My name is Julian Giuca and I was an early employee at New Relic, where I led their Logging product until 2022. It’s safe to say I have opinions about logs. Sometimes I think they are great. Too often I think we can do much better.
When people talk to me about logs, they reliably complain about observability costs. Engineering leaders have said it feels “like a protection racket”, where you're forced to capture and send everything with no real levers of control.
So I wanted to address this and built the ideal streaming pipeline to shape and route data. Basically Cribl for Datadog. If you can shape, filter, and route logs intelligently, you can slash your observability bill. Sending less data means spending less money. Coinbase had just spent $65M on Datadog (in 2023)! Surely, people wanted this… right?
Wrong.
Everyone talks about how much their observability bill is and what they’d like to do about it, but we found it's never a high enough priority. It was treated as the cost of doing business—a problem with no priority or owner.
SRE / DevOps are mostly concerned with keeping the lights on, not data management. There was some interest there, but minimal.
Platform engineers were interested, but were juggling other priorities. It also felt like this was an emerging segment of engineering, so it was hard to hone in on.
VP’s of Engineering were focused on execution and growing the top line. This meant observability costs were usually below the line in priority until uptime slipped or the CFO started screaming. A nice to have, but again, not enough of an urgent problem.
So, what gives? Cribl is crushing it, we see similar companies raising Series A’s, and we think we’ve got a better product. Turns out, they’re all security led.
We thought “reducing observability data to reduce observability spend” was the play, but no one owned the problem, so no one was incentivized to fix it.
Security teams don’t have that luxury. They need logs urgently. Cleanly. Ideally structured, and noise-free. They need signal out of all of these different sources and want to stop wasting time sifting through garbage data and alerts. SIEM vendors often serve the Dev market (I’m looking at you Splunk and Sumo), and I just spent too long looking at it that way—from the Dev lens. Couldn’t see the forest for the trees. Security teams are usually the ones driving adoption of these tools. Devs already have a wider selection of tools, and can take advantage of a logging platform.
I still believe that a pipeline is the missing layer—the third way—where you can structure, enrich, and route data before there’s a flood of alerts and you’re paying for data you don’t need. But I defer to you all.
Thoughts from folks here?
We have a really useful product for both Sec and DevOps, but cost isn’t a reason to try it.
So my ask is, try it out for 2 minutes: https://app.datable.io/auth/sandbox
It has all the features you’d expect, while being easier to get started and easier to use than any other pipeline on the market. We are actively working on automatic security detection and always adding more data sources/destinations.
Tl;dr: Built a product to address observability need. I misread the market; not enough urgency there. Discovered the real driver was SecOps.
One of the persistent challenges I run into in this area, is that any sort of up front filtering/routing requires you to know in advanced which logs are going to be important when an issue happens. Which is sort of impossible. And nobody wants to be the guy that filtered out some logs because they looked useless and then only later on realize they would have been instrumental in getting back up and running quickly.
One of the biggest problem we hear about from CISOs is 'they don't know what they don't know' - meaning they need a way to catch all the data. This plays pretty directly into your comment - there's a need for wanting everything, but a penalty for having everything - slower queries, expensive, more false positives, slower time to resolution.
What's common as a middle ground is blob storage and rehydration - where you send everything into low cost storage like S3 while still peeling off the high value data into the SIEM / Datadog / etc. Then if you notice something is amiss, you can rehydrate the time window you care about.
Kudos for being self-aware and acknowledging that solving the problem which you saw doesn't always translate into solving the problem that potential customers want to pay you to solve.
One of my favorite talks [0] speaks to the problem with thinking that telemetry is valuable just because it is [logs|metrics|traces].
Alerts/notifications etc. are an attempt to distill something useful from something that is abundant. From Cribl's `About Us` page [1] -- \ ˈkribəl \ - “An instrument with a meshed or perforated bottom generally used for gold panning in order to strain valuable material from discardable matter.”
Quite a few years ago, I led a migration off from a legacy logging provider that offered little more than full text search over unstructured text.
Logging at the time was somewhere in the ballpark of 1% of our total common infrastructure spend and widely acknowledged as too expensive relative to the minimal value we got from it with that rudimentary feature set, but it also was nowhere near enough cost to justify doing something about it. We had other observability costs that dwarfed it.
What finally justified the overhaul was that security couldn’t really operate usefully on log data unless we pulled the data out somewhere else like Athena and processed it there. That slowed down security incident response times dramatically.
The migration ultimately benefited the whole engineering organization but it had to be security led to get any traction.
Who are security teams? At which size company hires those? Is having a security team driven by copmliance to get certaion certificates that required by vendoes? Did the product you built addressed the observability need, if so, why it wasn't used much?
Are you pitching or complaining? Because I can't tell.
Mostly aiming to share my anecdote about being too close to a problem for others to learn from, and to pitch - to validate the security thesis.