This ESP32 antenna array can see WiFi signals! This is an unmodified smartphone, that's currently generating WiFi traffic. Now watch what happens when I take the image generated by the ESP32 antenna array, and I overlay it on top of the camera image: Now this phone turns into this glowing green blob. This is not just a simulation. If I start walking around, you can really see how the ESP32 antenna array can visualize the WiFi signals that this device emits! Metallic walls like this one here behave like mirrors for radio signals. You can see the direct line of sight path, but also the reflected path in the mirror. You might also be able to tell that the reflected path has a very slight yellow-ish tint, this is how I visualize the path delays and the path delay of the reflected path in the mirror is a bit higher than the direct line of sight path. Especially if I block the direct line of sight path with my hand, or at least attenuate it, you can really see that now you're only receive the reflected path. Of course, for mirrors, we know that angle of incidence is equal to angle of reflection, so therefore, whether or not you can see the reflection depends on my location. So, if I come closer, the reflection first moves a bit, and now you can only see the attenuated line of sight path. Whereas here, if I'm standing here in the back, now you can again see both the reflected path and the line of sight path. Non-metallic walls only attenuate the signal, so devices remain visible on the camera image. As I'm entering the room next door, you can clearly see me moving around behind the wall. The software filters packets by MAC address, so you only see my device, even if other WiFi transmitters are nearby. Highly directional antennas, like this Yagi antenna, focus all of the transmit power into a very narrow beam, so they behave a bit like a flashlight for RF (radio frequency) signals. This means that if I point this antenna directly at the ESP32 antenna array, you can see how the webcam image becomes totally overexposed. You can also see the signal that this antenna is emitting, if I point it at some reflector like this metal wall here. Here you can also really nicely see the yellow-ish tint which is due to the higher path delay. On the other hand, if I take this antenna and point it somewhere else, you can hardly see anything on the WiFi image, and this is because only very little power reaches the ESP32 antenna array from this Yagi antenna. To some degree, the antenna array can also track passive targets, like whatever this is. Reflective materials like the tinfoil in this scene produce a bright spot in the image, non-reflective targets can be seen as shadows in front of previously existing propagation paths. This way, the array can act as a passive radar system by exploiting existing WiFi signals like SSID broadcasts from an access point. Outdoors, propagation paths tend to be longer. With 40MHz of bandwidth in the 2.4GHz WiFi spectrum, we can achieve some level of depth perception. Path delays are indicated by color, with the varying reflected path lengths creating this rainbow pattern. Shorter ground reflection paths appear in purple or blue, while the late-arriving reflection from the building in the background is shown in red. So how does all of this work, how can we get this kind of information from a cheap WiFi microcontroller like the ESP32 that you can get for less than 1 euro 50? Let's go back to the scenario with the metal wall from earlier. In this somewhat oversimplified explanation, the transmitter is modeled as a point source emitting an electromagnetic wave. When the transmitter sends out the WiFi packet, the ESP32 antenna array first observes a wavefront from the direction of the source, and, slightly later, a wavefront coming from the reflector. Depending on the angle the signal is coming from, the individual antennas observe the wave that is moving at the speed of light at slightly different points in time, with the time difference measured in picoseconds. To determine the angle of arrival and to produce an image from the received WiFi signal, we need to synchronize the receiver chains inside the ESP32s precisely. In other words, we need to achieve something called phase coherence. Typically, a relatively inaccurate crystal oscillator serves as the frequency reference for the radio receiver inside the chip. To synchronize multiple ESP32s, we need to provide the same 40MHz frequency reference to all receivers. This ensures that the local oscillator signals across chips are now frequency-synchronous. For phase coherence, we also need to make sure that the local oscillator signals are synchronized in phase. Providing the same frequency reference to all ESP32s is insufficient for that. To understand why, we need to take a closer look at the internal structure of the chip. Here, we find something called the PLL (phase locked loop) that derives the local oscillator signal from the 40MHz reference. The local oscillator is then used to mix the WiFi signal down to baseband. Let's observe what happens to the output of the PLL as we turn the reference signal off and on again. While the local oscillator signal is always at the correct frequency, we get a random phase offset relative to the 40MHz reference. This effect is sometimes called phase uncertainty, and it is likely due to the specific kind of PLL used inside the ESP32. To get rid of this offset, we need to add another ESP32 to the system, which provides phase reference packets. These packets travel across microstrip lines on the back of the PCB, with well-known length and group velocity. By measuring the phase of the reference packets at each receiver, we can compensate for the phase uncertainty in software. This calibration procedure is only necessary once after each ESP32 has booted up, or in case we switch the WiFi channel. The esp-idf WiFi driver documentation lists all steps needed to obtain the so-called channel state information for every incoming packet. Channel state information from a single ESP32 looks like this: We get a phase and an amplitude measurement for every subcarrier inside the WiFi bandwidth. You can see how the curves change if the transmitter starts moving. This is the channel state information from all eight ESP32s in the antenna array. Each color belongs to a different antenna. By taking the Fourier transform along the subcarrier axis, we obtain the channel impulse response. The x-axis can be interpreted as a time axis now, and the y-axis indicates the amount of power received at different points in time over different propagation paths. We can now apply array processing algorithms to determine the angle of arrival of the incoming signal, or to generate the WiFi camera images. With multiple antenna arrays, we can even build an indoor navigation system. Combining the angle of arrival estimates from multiple points of view, we should be able to triangulate the location of the signal source. This animation shows the triangulation results from a measurement campaign with a transmitter on top of a robot. If we provide the same clock and phase reference packets to all four ESP32 arrays in this system, we can also use time and phase difference of arrival. Obviously, the best localization accuracy is achieved, if we combine both techniques. By supplying a shared clock and phase reference to multiple arrays, as mentioned in the TDoA scenario, we can also combine them into larger arrays. In practice, phase-balanced RF splitters at the back of the antenna array distribute the reference signal, ensuring phase coherence across all antennas in the combined array. Larger arrays offer more accurate angle of arrival estimates and a higher resolution for the WiFi image. One more remark on indoor localization: Triangulation and TDoA-based trilateration assume multiple arrays and the existence of a clear line of sight path between transmitter and receiver. But what if there is only a single array and obstacles in the environment? Can this antenna array still determine the transmitter's location on the other side of the metal container? By now, you have seen how radio signals reflect off various surfaces like the floor, the ceiling and walls. By observing one or more moving transmitters for a while, we can collect a dataset of wireless channel measurements. We can then feed these channel measurements into a neural network using a self-supervised dimensionality reduction technique called Channel Charting. After some neural network training, this method figures out the geometry of the environment. In other words, even a single array can locate transmitters behind corners by observing reflected waves. The resulting map, called the channel chart, is fairly accurate in both line of sight and non-line of sight areas. Once trained, neural network-based localization is possible in real time. More information, resources, and channel state information datasets are available of the project website.