Google's Offline Personal Assistant

In June 2015, there were some first rumours that few commands for Google Now would be available even when you are offline. An APK teardown of the Google app reported at
http://www.androidpolice.com/2015/06/27/apk-teardown-google-app-v4-8-prepares-for-ok-google-offline-voice-commands-to-control-volume-and-brightness-and-much-more/ revealed some string resources that hinted on this.

<string name="offline_header_text">Offline voice tips</string>
<string name="offline_on_start_cue_cards_header_listening">Offline</string>
<string name="offline_on_start_cue_cards_header_timeout">Offline voice tips</string>
<string name="offline_on_start_cue_cards_second_header_listening">You can still say...</string>
<string name="offline_on_start_cue_cards_second_header_timeout">You can still say "Ok Google," then...</string>
<string name="offline_on_start_cue_cards_second_header_timeout_without_hotword">You can still touch the mic, then say...</string>
<string name="offline_options_start_hotword_disabled">You can still touch the mic, then say...</string>
<string name="offline_options_start_hotword_enabled">You can still say "Ok Google," then...</string>
<string name="offline_error_card_title_text">Something went wrong.</string>
<string name="error_offline_no_connectivity">Check your connection and try again.</string>

So far,  Google Now required an online connection to work. Since this is not always a given it is beneficial to have a workaround in these cases. They found the following four options:

• Make a call
• Send a text
• Play some music
• Turn on Wi-Fi

Usually, such a tear-down is more of the kind of a rumor than facts. In this case, these rumors proved to be true. In September 2015, this functionality was made available as reported, e.g., on http://www.androidpolice.com/2015/09/28/the-google-android-app-now-supports-limited-voice-commands-for-offline-use/. The way this is reflected on the UI is shown on the following picture

Google Now in offline mode,taken from http://www.androidpolice.com/2015/09/28/the-google-android-app-now-supports-limited-voice-commands-for-offline-use/

So, some commands are also available, when you are offline. So far, this list is larger than the original one, but limited to

• Play Music
• Open Gmail (works with any app name on the device)
• Turn on Wi-Fi
• Turn up the volume
• Turn on the flashlight
• Turn on airplane mode
• Turn on Bluetooth
• Dim the screen

Unfortunately, this is only true for the English version. For instance, it refuses to work in German, even if the English offline recognition is downloaded to the device. Neither English or German commands will work. Instead, the following screen is shown.

Google Now in offline mode for German on my Samsung Galaxy 5

Yet, it is unclear, when this will work. But, Google seems to be advancing their embedded technology as reported in Personalized Speech Recognition on Mobile Devices. Here, they describe a remarkable speed-up of their embedded speech recognizer. They state their newest technology "...provides a 2× speed-up in evaluating our acoustic models as compared to the unquantized model, with only a small performance degredation". The recognition performance for open ended dictation in an open domain WER increased from 12.9% to 13.5%. Moreover, they report a decrease of the footprint. Their acoustic model "...is compressed to a tenth of its original size". Apart from that they still feature language model personalization through a combination of vocabulary injection and on-the-fly language model biasing.

In the end, they "built(d) a system which runs 7× faster than real-time on a Nexus 5, with a total system footprint of 20.3 MB"

The latter work only aims for the recognition task as it is available with what they call "Voice Typing". It still needs integration of NLU to make it actual commands to use it for Google Now.

So, Google seems to be on the way for a personal assistant that can also be used if you are not connected to the internet. Some of the commands may make not much sense if you are offline, but some will work, sooner or later. English is supported in first place and it is unclear when other languages will follow. But it is a start.