Home > News content

Cambridge University's New Research: Using AI Algorithms to "Monitor" Mobile Phone Typing

via:博客园     time:2019/4/2 18:40:22     readed:233

Chestnuts and Anne are from Aofei Temple. Quantum bit is produced. Public number QbitAI

Touch screen can also sell your mobile phone password.


Mobile phone built-in microphone, with AI algorithm, can make every word entered nowhere to escape.

You may not hear it, but every time your finger taps on the screen, it emits a sound wave. Microphones less than 1 centimeter away from the screen can be easily recorded.

Scientists at Cambridge University have developed an AI for eavesdropping on touch screens. It can restore the human input information from the subtle sound.

Sound wave betrays you

Looking around when nobody is there, quietly unlock a lock on the touch screen. Why is the password leaked by the sound wave?

Even if you operate quietly on a touch screen, every slight tap of your finger produces sound waves. When the screen is jammed at different positions, the acoustic information will change accordingly.


Yes, key information is hidden in these different shapes of sound waves. This information can be captured by the built-in microphone of the mobile phone.

Researchers have developed a mobile malicious application backhand, which, when implanted into a mobile phone, can mobilize the built-in microphone, let it work secretly, and record the sound information when touching.


DeltaMalicious Sound Wave Collection Program Used in the Experiment

They collected the corresponding information between the shape of the sound wave and the position of the touch, trained an AI model and restored the input content.


So far, the crime process has basically taken shape:

AI, together with microphones, has a clear division of labor, should fit in and out, and steals your privacy in an orderly manner.


Field verification

How effective is this program? The researchers recruited 45 volunteers and began a series of tests.

The Nexus 5 mobile phone and Nexus 9 tablet are selected for the test. The former was launched in 2013 with a screen size of 4.95 inches, while the latter was released in 2014 with a screen size of 8.9 inches. Both devices have two microphones built in.


DeltaThe microphone position of the experimental equipment indicates: Nexus 5 on the left and Nexus 9 on the right.

In order to simulate the real world environment, the researchers did not choose a quiet laboratory, but chose three daily environments to test the accuracy of the whole system in different noise environments:

  • Public Lounge: People are chatting around and occasionally make loud noises when making coffee.

  • Reading room: Computer keyboard sound mixed with whisper voice environment.

  • Library: Almost no voice, but computer keyboard sound surrounds.

In these three environments, 45 volunteers were asked to conduct four experiments.

The first group of volunteers needed to enter numbers 1-9 randomly, 10 times for each number, 200 groups of four-digit passwords for the second group, three groups of random letters for the third group and five-letter words for the fourth group, all from the chat corpus NPS.

The experimental results verify the feasibility of this attack method.

In 20 experiments, AI can accurately restore 91 out of 150 four-digit ciphers with a reduction accuracy of 61%.



It's not a problem to crack the symbols of letters. When 27 words were tested, only 10 experiments were carried out. The model cracked 7 words on the mobile phone and successfully restored 19 of them on the tablet.


That's it. You don't notice it, but the passwords go all the way.

I have a lot of little mimi, all in sound waves

Actually, there is no need to panic, because AI wants to get the mobile phone password, and this is not the only way.

Last September, a study published by the University of Lancaster in the United Kingdom stole passwords, especially graphical ones, using sonars in mobile phone speakers.


The principle of sonar is to calculate the time difference between the sound wave sending out and returning to determine the position of the object and whether the object has moved. In this study, AI analyzed the echoes recorded by microphones to track the movement of a user's finger on a smartphone screen.

It's also when you're not aware that everything happens.

Of course, these are just mobile phone problems.

Nowadays,From keyboard to hard diskWhich one can't hear your secret?

Please note that this is a serious statement, not driving. )

Even the potted plants at home may know what you said.

So let's talk about it one by one.

Listen to the keyboard first

Last December, there was a name.KeytapKeyboard eavesdropping, simple and effective, has attracted a lot of attention.



First use the microphone to collect the sound waves of keystrokes, such as three times for each key.

Then a simple prediction model is built to calculate the average waveform of each key:

The collected peak values are aligned to avoid the effect of delay; the waveforms are aligned precisely with Similarity Metric; after alignment, the average waveforms are obtained by making a simple weighted average.

Finally, tap the keyboard to your heart's content, as long as you compare the similarity with the average waveform, you can detect which button it is.

And open source code, you can play directly.

Hard disk eavesdropping, microphone not needed

This is the crystallization of the brain holes of the University of Michigan and the University of Zhejiang. The original mechanical hard disk, without microphone, without any hardware changes, becomes an eavesdropper, and the sound quality is good:


When the hard disk is working, the spindle runs at high speed and drives the disk. The magnetic head on the disk will sense the change of magnetic field on the disk and write data by changing the magnetic field.

This process is very delicate. As long as the magnetic head is impacted by external sound waves, the head will be offset. The displacement is reflected in the voltage signal generated by the position sensor of the hard disk.

Ask AI to analyze this slight deviation. Scientists have restored what humans say and high fidelity music. Shazam can also answer the title of the song by listening to it.


After all, the sampling rate of the hard disk is over 30,000 hertz, which is almost CD-level recording quality.

Potato chips, not just crisp

Elegant voice may be the greatest pleasure in eating potato chips.

But to eavesdrop on human speech, scientists do not rely on the sound waves generated by the fragmentation of potato chips, but visually move their minds.


Mixed teams of MIT, Microsoft and Adobe use high-speed cameras to shoot the vibration of potato chips through sound-proof glass. The algorithm can judge whether the speaker is a man or a woman, or even restore the content of the speech.

Researchers say that touching objects around when sound travels creates a subtle visual signal in vibration that the naked eye cannot detect, but is captured by high-speed cameras (2000-6000 frames per second). AI can recognize human conversation by analyzing visual signals.


Moreover, not only potato chips, but also aluminum foil, water cups and even potted plants can be used for eavesdropping. These objects appear in the room, human beings will not be surprised at all, it is really unique.

In this study, the top SIGGRAPH 2014 was selected.

So, maybeEverything in the worldEveryone knows your secret.

Fortunately, these methods are still in the research stage.


Four researchers, Ilia Shumailov of Cambridge University, Jeff Yan, Ross Anderson and Laurent Simon of Samsung American Research Institute, were involved in this paper.

Paper Hearing your touch: A new acoustic side channel on smartphones address:


China IT News APP

Download China IT News APP

Please rate this news

The average score will be displayed after you score.

Post comment

Do not see clearly? Click for a new code.

User comments

Related news