Over 20 years ago I had a Toshiba Tablet PC convertible that had a beam forming array of microphones and it came with software that let you point where you wanted to record from.
The use case was for lectures, you could tell the laptop to just record from behind it, pointing the beam in the direction of the professor.
Amazing idea and something I haven't seen since.
In the golden age of mini camcorders, some Sony Handycams had "zoom" microphones which used beam forming to limit gathered sound to roughly the area equal to the what your sensor sees.
Another great idea.
Oh. They still make similar stuff: https://electronics.sony.com/imaging/imaging-accessories/all...
I feel like my iPhone does it. But not sure. Sound definitely changes when you zoom while recording
They do. They rarely mention it but they do:
https://devstreaming-cdn.apple.com/videos/wwdc/2019/249a0jw9...
The only content regarding audio I saw here are slides 124-140, which cover beam-forming but I didn't see anything about a default beam-forming profile tied to virtual zoom.
On current iPhone Pro (16) you can even select the audio mix you want for recorded video after recording.
This is a feature of iPhone, yes. Believe it came around the 11 (?) but it can really help when recording concerts if you're into that sort of thing.
Funny, that’s exactly when I hate it the most! If you zoom mid clip the sound very audibly changes which is not desirable.
Samsung phones have this as well, can be enabled or disabled in the camera settings.
Mine is too old to test the claim, but knowing that it has at least three microphones on board, It'd be absurd if Apple didn't implement it.
It's pretty computationally cheap, too, as long as you've got the math right and an easy way to choose where to aim the beam
It's used widely in fancy videoconferencing setups.
The mic array for the room figures out who's talking and isolates the audio from them.
(Videoconferencing in large rooms has long picked the loudest microphone to use at any time, to avoid mixing in noise from other mics, but then the beamforming makes it that much better.)
I'm wondering if that's why those kinds of setups offer good audio if only one person speaks and there's clear pauses between people, but as soon as you have a quick back and forth or two people talking the audio turns into complete mush.
I wonder how that worked. Assuming the microphones were on the screen plane rather than the body, it wouldn't be able to tell the difference between "straight in front" and "straight behind".
Straight in front is likely to be unobstructed while straight behind is likely to be obstructed by computer hardware. Therefore, straight in front is likely to have crisp sound while straight behind is likely to be muffled and/or distorted by reflections.
Yes, but that's not something you can beamform away with a planar array. Especially if your goal is to record what's going on behind the screen. You need something out of plane. Which they may have had! I don't know the details of the hardware.
As someone who made the mistake of putting a webcam cover over the tiny little microphone hole above the screen (it picks up very little besides impact noises now), it wouldn't be hard to have a mic hole facing in both directions to solve that problem
A single mic facing in both directions, or one mic next to the other but facing opposite directions, doesn't really help. You need separation between them in the direction of wave propagation (so in the front-back dimension of the laptop screen in this example) to tell which direction the sound is coming from.
They're in somewhat random locations, not symmetric and parallel as one might expect.
Sennheiser has a model that is mounted on ceiling. Haven’t seen this live.
https://www.sennheiser.com/en-us/catalog/products/meeting-an...
The attenuation provided by the tablet case / shell is quite significant. I bet they had some extra foam, or, something, to make it even stronger. So the "right behind" signal would be heard only if "right in front" is not readily drowning it.
Idea I've had for years but never got around to testing due to lack of compute:
Use a microphone array and LIDAR for ground truth, and train a diffusion model to "imagine" what the world looks like conditioned on some signal transformations of the microphone data only.
Could be used by autonomous vehicles to "see" pedestrians through bushes, early detect oncoming emergency vehicles, hear bicyclists before they are visible, and lots of other good things.
This already exists, it's the domain of inverse problems. Inverse problems consider a forward problem (in this case wave propagation) depending on some physical parameters or domain geometry, and deduce the parameters or geometry from observations.
Conceptually, it's quite simple, you need to derive a gradient of the output error with respect to the sought information. And then use that to minimize the error (= "loss function" or "objective" depending on field terminology), like you do in neural networks.
In many cases, the solution is not unique, unfortunately. The choice of emitters and receivers locations is crucial in the case you're interested in.
There's a lot of literature on this topic already, try "acoustic inverse problem" on google scholar.
So basically a kind of passive echolocation?
I like it. I think you'd need to be in known motion around the area to build up a picture -- I don't think it would work with a microphone just sitting in place.
Sort of!
If you shut your eyes off and you hear footsteps to your right you have a good idea of exactly what you're hearing -- you can probably infer if it's a child or adult, you can possibly infer if they are masculine or feminine shoes, you can even infer formal vs informal attire based on the sound of the shoes (sneakers, sandals, and dress shoes all sound different), and you can locate their angle and distance pretty well. And that's with just your organic 2-microphone array.
I imagine multiple microphones and phase info could do a lot better in accurately placing the objects they hear.
It doesn't need to build an accurate picture of everything, it just needs to be good at imagining the stuff that actually matters, e.g. pedestrians, emergency vehicles. Where the model decides to place a few rustling leaves or what color it imagines a person to be wearing is less relevant than the fact that it decided there is likely a person in some general direction even if they are not visible.
I just think diffusion models are relatively good at coming up with something explainable and plausible for a given condition, when trained on some distribution of data.
Like "oh I hear this, that, and that -- what reality could explain those observations from the distribution of realities that I have seen?"
Sounds like passive radar moved to the accoustic domain. It's a neat thing and there is some open source work around it. However it's also a good way to run afoul of ITAR, passive radar is still one of those secret sauce, software is a munition type things.
I have a passive radar. It also is a direction finding radio. I didn't have to jump through any hoops.
On recent devices with on-device NPU, could be combined with RF imaging of nearby activity and structure via WiFi 7 Sensing Doppler radar.
From the samsung S10 forward, this is a feature while recording video in zoom mode. I was always really curious how they did it.