He doesn't mention how he calibrates the camera extrinsics or intrinsics. This isn't new, it's basic structure from motion. He is going to be amazed when he "discovers" photogrammetry and bundle adjustment next.
I checked out the code and there is no consideration for camera parameters it seems. It's a neat demo but impractical given the need for precise camera calibration over distance. Long baseline stereo has problems unless you can figure out how to keep the cameras aligned within fractions of a mm over great distances.
At 2:39 he talks about camera orientation and says that he has a solution.
What he is doing isn't long-baseline stereo imagery? And clearly is not aligning the cameras: I'm guessing he is ensuring they don't wobble and he is solving the camera orientation outside of the code you can see (maybe using reference objects - sun or daytime moon?).
I enjoyed his ponderings on bird-strike avoidance - maybe commercialisable. The asteroid detection seemed way more farfetched (light gathering is hard).
In order to project the rays into the voxel grid and have them intersect when they are projecting the same object, the cameras do need to be calibrated. This is very much multi-view stereo CV. I’d guess that he’s taking nighttime photos and comparing with a ground-truth star map and choosing correspondences to compute a rotation/translation matrix for each camera.
At night in Ukraine they use microphone arrays to track the drones:
https://www.twz.com/land/thousands-of-networked-microphones-...
* Locating moving objects that could be anything that can move at a speed, with Cheap Cameras Without Using AI or Radar
Aerodynamics & engineering dictates that not many things can move at the speed/agility of a stealth fighter.
If you've got a radar system already in place, "Stealth" is just the ones that don't also pop up on the radar.
Presumably scattered radar receivers can detect faint reflections using similar correlation.
Is there a background microwave radiation at shorter wavelengths that could be made into a camera that could see through clouds too? microwave is too long wavelength to make a small camera.
> Presumably scattered radar receivers can detect faint reflections using similar correlation.
Yes, multistatic radars are already in service. An advance on the concept is that the active transmitter can be placed outside anticipated weapons range and those within range are merely passive receivers.
There have been proposals to build such systems that use 'oblivious' transmitter sources such as commercial radio broadcasts.
There was a project to extend/enhance the Kraken multi-SDR system that got yanked from GitHub, etc, because it tripped over ITAR in that it enabled passive radar.
ITAR assumes the US leads the world in technology, and actively prevents us from catching up if we don't.
Locating civilian, friendly and radar-visible aircraft is a solved problem though. So the distinction is only a matter of subtraction.
No, in fact this kind of technique absolutely puts a hard limit on the value of "stealth" devices, which are purpose-designed to evade a very specific technology that was a warfare-changing innovation in the 1940's but is now pretty cheaply replaceable by consumer junk.
The days of large manned aircraft are very much numbered, and armies that fail to see that are armies that are destined to lose.
Nah. That's such a naive and uniformed take. Signature reduction (stealth) remains broadly useful in that it makes air defense harder and more expensive. Anything based on cameras using ambient light is going to be of very limited use, especially at night or bad weather. And cameras can't be permanently installed on the open ocean.
The days of large aircraft are in no way "numbered". In order to have enough range to operate in the Indo-Pacific Theater that requires a lot of fuel, which implies a large aircraft (regardless of whether it's manned or unmanned). And while AI might eventually be effective at controlling combat aircraft, as of today that remains science fiction. So that means you need a person either directly in the combat aircraft, or actively controlling a large "loyal wingman" type drone from within line of sight (to ensure reliable communications in a hostile EW environment).
Indeed. And to add to your argument, AI may be effective in controlling aircraft for certain tasks some day. But there are still many that will either require a pilot there, or greatly benefit from a pilot there.
The "drones will replace everything" argument does not understand fully the missions required of combat aircraft. I won't be as foolish to say it could never happen. But to those who argue that, say, the F-35 should be canceled in favor of a drone, need to show how their drone can do what a manned strike aircraft like the F-35 can do.
And indeed, stealth was always a "make air defense more expensive" prospect. And while techniques have come out to counter some of the benefits of stealth, they all come with tradeoffs and added costs....which was precisely the point.
The question isn't whether One Specific Mission might be replaceable. It's whether a war fought with the 1000 F-35's needed to do that mission can be successfully prosecuted against an enemy armed with A Million Drones (because yes, 1000x is just about the cost delta we're talking about).
Consider: Germany in 1940 had absolutely no answer for French heavy armor. There were weapon systems along the Maginot line which the wehrmacht couldn't counter, and everyone knew it. They didn't need to.
You're not even asking the right question. There's nothing magic about cheap drones. They can be effective supplements to crew-served weapons and light artillery for close combat. But they lack the range to be effective in the Indo-Pacific Theater, and when drones are designed with sufficient range they stop being cheap. Besides the range problem, cheap drones also can't perform the full spectrum of missions like deep strike, ASuW, interception, etc.
For all its faults, the F-35 at least has sufficient combat radius and survivability to be relevant in a major near-peer conflict (although it desperately needs a new and more expensive adaptive cycle engine). Until a better platform comes along the choices are essentially the F-35 or nothing.
> And cameras can't be permanently installed on the open ocean.
If the camera can self-calibrate against known objects then camera motion can be adjusted for. Could use visible reference points (sun, moon, stars) with clear skies. Stars may be visible by a sensitive camera in daytime with good collimation (bottom of well story https://www.snopes.com/fact-check/star-search/ ). There are also radio sources (GPS satellites) or more SciFi solution of noisy stellar objects??
> And while AI might eventually be effective at controlling combat aircraft, as of today that remains science fiction
That's ridiculous, sorry. There are literally AI bots being banned every day on War Thunder! There are armed long-range drones under cross-continent satellite control already! Of all the things AI/automation might have trouble with, operating rigorously-documented aircraft systems and making shoot/don't-shoot decisions of a handful of targets is surely one of the easiest[1]. If you as a military planner don't do this because you think it's impossible, you're going to lose.
[1] I mean seriously: would you have an easier time making AI to fire a missile or sew a prom dress? Not even close, the dress is harder. And AI garment manufacture is reaching the market already!
Nah. That's such a naive and uniformed take. War Thunder is a highly simplified toy video game. It has no relevance to the real world.
Satellite control is also irrelevant. In any major near-peer conflict the reconnaissance and communications satellites will be the first casualties. The USA and China are currently engaged in an ASAT arms race. That's why they're back to planning and training to fight without satellite support.
Games are not remotely as complex of an operating environment as the real world.
If it's a clear day and the bottom of the airplane is painted blue, it could blend in with the sky. For that matter, didn't Wonder Woman have a transparent airplane?
Aha, better yet, it was invisible! https://dc.fandom.com/wiki/Wonder_Woman%27s_Invisible_Plane
Yes, because if there's one thing computer vision solutions have historically had trouble with, it's being fooled by paint. The ludditism in this thread is shockingly strong.
CV systems absolutely do struggle with this...
Yes, and that's considered a bug to fix, and can be. CV systems are widely applied to find camouflaged and obscured objects! Literally every automated wildlife camera is proof that this is a silly point.
Call it whatever you want, lack of contrast is a physical constraint on your system and can be trivially leveraged. There's been an arms race between optical camouflage and detectors for the entire history of eyes.
It's not necessarily a game ender, but it's not something you can just wave away either.
He didn’t go into detail on how the cameras are calibrated by I wonder if he’s using stars to do that or what. Standard SfM would struggle with such featureless images and wide baseline.
I thought: airplanes, the sun, the moon could all work
You need a lot more than one reference though. At minimum I think you need 8 correspondences to solve for the camera’s pose. Things like airplanes could work if you had enough of them, but then the imagery in all your cameras would have to be shutter synced, which is impractical with a bunch of web cams
Is anyone else here interested in actually trying this? I live near a fairly active small airport in Lansing, Illinois, and in the flight paths of everything going to/from Midway and Ohare, so combining this with ADSB might make for some interesting sources of calibration data, and a way to debug this algorithm.
I think a neighborhood of Raspberry Pi based cameras networked together could prove quite effective. I wonder if all the installed Ring door cameras could do this in a national security emergency.
I further wonder if it's being done to the US (with the current installed camera base) by China or others.
* as long as it's a bright sunny day
Good look doing that at night or when it's cloudy.
I'd also be curious about the detection range, even on a sunny day.
Stealth doesn't mean the aircraft is invisible to radar, it just means it has a very low radar cross section, so you can't detect it from very far away. Fly close enough to an air defense radar, it will reflect enough energy to show up, it's not a black hole. The point of stealth is to make it impractical and uneconomical to build an air defense system with a high enough density of expensive high performance radars to reliably detect the aircraft.
The video says up to 100km. With a jet moving at mach-whatever and air-to-surface missiles they can launch from 200+km out, not super useful in practice, even if it worked at night or on cloudy days.
Ultimately this is an economic argument. With about $10k you can buy a few dozen machine vision cameras [I would say "webcams" if I knew less about webcam compression], a PC with mobile modem, and a self-contained solar power supply. For $10M you can buy a thousand of these. A thousand of these evenly distributed within a few hundred kilometers of your country's borders are going to add dramatic capability to your air defense network compared to going from 5x $10M radar systems to 6x $10M radar systems. The key bit is that $10k costs far, far less than a single strike missile to take out one of these optical detection sites.
A full Patriot battery is $1100M, with the durable radar and command components being more like $400M.
Add another $10M and build passive radars from consumer stuff that listens for ruccus in telecom and television fields and you have a pretty annoying defensive baseline.
few old iphones or android smarthphones would do the trick even better - they have much better cameras than webcam, old iphones can even record at 240fps. NPU, GPU, LTE, GPS, battery, IMU, stereo microphones also already included in one package.
not really as most modern airborne weapons platforms operate bvr, even if they were on the horizon, and your array of cameras were operating in optimal conditions there is a still a hard limit on the effectiveness of such a solution.
For offensive work, yes. If you control the ground, though, and are policing a defensive perimeter against air attacks on targets in the interior of your territory, you can make and distribute the camera stations cheaply enough that it doesn't make sense destroying them.
Could the advantage be that a military monitoring installation of cameras is less detectable than a high-frequency radar monitoring installation? My understanding is that active radars are easily detectable, but perhaps cameras could be less so and that advantage could allow them to be deployed closer and might require less protection themselves.
Yup, that's certainly one. I see this technique as simply one more tool in the toolbox. Air defense (and really, all of warfare) is not a single "this thing is better than the rest" but a system of things that work in conjunction with one another. This system has advantages (passive, cheap, portable) and disadvantages (short range, clear weather only). It's good and useful work, and could conceivably be part of a larger system.
Most stealth aircraft are only stealthy against the higher frequency radars.
And only the higher frequency radars are really useful for interception or weapons targeting. The lower frequency radars are good at long-range detection but they're far less precise.
Early stealth aircraft designs like the F-117 and B-2 were optimized for signature reduction in the frequency bands most typically used by air defense and fighter radar sets. Newer designs like the B-21 and F-47 supposedly have "broadband stealth" which reduces radar cross section across a broader range of radar bands but details are classified.
The "stealth aircraft hate this one weird trick" conversation also has to contend with jamming, because many of the techniques that work great at spotting stealth aircraft in peacetime are especially vulnerable to jamming. This makes them far less useful in an actual conflict.
The Saab Gripen is said to be particularly effective in this regard.
I'm pleased at the discussion in this chain of replies, as it illustrates the cat-and-mouse game stealth (low observability?) is. Stealth never was a perfect defense, as it's detractors claim. It forces the adversary to make tradeoffs, such as increasing the costs needed to detect such craft, which can then be exploited for gain.
Jamming is a function of power and there are few radars that put out more power than the low frequency surveillance radars. Duga was colocated with the chernobyl nuclear power plant.
Aren't some folks just watching the reflection of cell phone signals? Why bother with an expensive radar system when the cell phone system is flooding the sky with radiation anyway?
Depending on the war, you might not have cell phone (basestation) signals everywhere during actual fighting, relying on that seems a bit limiting.
Sure because at end of day, radar is all about detecting bouncing radio waves.
However, it's accuracy of detection that changes the battlefield and my guess is detecting disturbed cellular signals gives very low detection chance/range.
Also, if enemy thought it was problematic, hit cellular tower with missile is extremely easy.
Cells are highly directional, and should ideally not be installed facing the sky. Add stealth coatings to the mix, and your return signal may not be enough to detect. Radars use high power emitters.
https://www.rcrwireless.com/wp-content/uploads/2023/08/Pictu...
Super cool! Though cloud cover disabling your detection technique may make this infeasible for anything but low-altitude drones.
One small nit: Calling Ray-tracing "Not AI" is like calling Matrix Math "Not AI". It's definitely a core component of a lot Machine Perception (i.e. AI) tasks.
I'm guessing OP meant "Without using any costly CNN or Transformer inferencing", which is actually a clever achievement.
If you spam a dense enough network of connected sensors you can detect anything.
> This technique can also be used to detect drones from many kilometers away
Sudden spike of git pulls from Ukraine for the repo...
The ukrainians are very good at this already. Since drone swarms typically arrive at night they tend to prefer sound sensors.
