Pentelligence

Pentelligence is a novel interaction device that does not require special paper or additional hardware (except for a computer) to recognize handwriting. This pen can achieve recognition rates of over 98% for digits on individual users without writing instructions. Its main strength lies in the use of deep neural networks that learn the individual writing styles of the users. By combining motion and audio data from inside the pen, the strengths of the individual sensors complement each other.
On this page we offer a short introduction to our project and give a brief overview of the main features of Pentelligence. For further information, please refer to our paper: Pentelligence: Combining Pen Tip Motion and Writing Sounds for Handwritten Digit Recognition

Developed by the Human-Computer Interaction group at the University of Hannover.

For thousands of years humans wrote down their ideas in various ways. Even today, the pen is an indispensable part of our lives. Although even tablets offer precise input options, many people still prefer the pen and paper.

The vision of Pentelligence is to bridge the analog and digital worlds with a device that doesn't differ from the appearance of ordinary pens. In our paper at the CHI 2018 we have shown the concept and the recognition of digits, which is particularly important because, in contrast to words, in many cases no subsequent autocorrection can be performed.

The project will continue to pursue the vision of a digital pen that does not require special paper, any other additional hardware or even cameras to recognize handwriting.

From the inside, Pentelligence relies on small, inexpensive and easily available hardware components. With a length of 12cm and a diameter of 12mm, the prototype almost reaches the dimensions of a conventional ballpoint pen in the first phase.

A microcontroller with an easy-to-program Arduino bootloader records the measured values of the microphone and the inertial measuring unit together with a specially self-developed binary writing pressure sensor and sends them to the computer via USB.

In future releases a wireless version will be developed, which will allow an even more realistic look and feel compared to ordinary pens.

Handwritten digits are as unique as their writers. For this reason, modern methods of deep learning are necessary to customize Pentelligence to the individual writing style.

The envelopes of the writing audios, fed into the neural networks, eliminate surface-related features. Downsampled and averaged motion data together with write pressure information capture individual strokes and reconstruct the writing trajectories within the classifiers.

By subsequently combining the results of the audio and motion classifiers, the complementary features can be merged and recognition rates of over 98% can be achieved.