Smart Interaction Beyond the Smartphone

A similar opinion as stated in my last blog about Smartphones as Explicit Devices do not meet Weiser's Vision of Ubiquitous Computing stated Jennifer Winter in a blog  for user testing. In her blog Will AI Replace Your Smartphone? she bemoaned the bad user experience of smartphones requiring their users to pick up their phones, start an application to actually go for what they really want.

She sees more user friendliness in those interactions that disappear (similar to Weiser's vision). And there is some potential in it, despite the fact that users become addicted to their smartphones. Jennifer Winter mentions a statement from the Mobile World Congress in Barcelona in March 2015, that 79 percent of smartphone users have their phones within arm's reach for all but three hours of the day.

However, users see alternatives to that as a study by the Ericsson's Consumer Lab revealed.The study states: "1 in 2 smartphones users now thinks that smartphones will be a thing of the past, and that this will happen in just 5 years." Users want smart interaction with objects, but it does not need to be mediated by the smartphone. The study reveals even more potential about the use of artificial intelligence in our lives:

• 85 % think wearable electronic assistants will be common within 5 years
• 50 % believe they will be able to talk to household appliances, as they do to people
• ...

More statements are in the following figure of that study.

Consumers who think using artificial intelligence (AI) would be a good idea 

It is unfortunate. that the study does not differentiate between AI and its interface. And so does Jennifer Winter. The technology behind the stuff we have with smartphones today is for sure artificial intelligence, cognitive computing, ... The difference lies in the interface which should (and will) disappear in the next few years.

Smartphones as Explicit Devices do not meet Weiser's Vision of Ubiquitous Computing

Mars Weiser uttered the vision of the invisible computer as "The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it." He mainly based his vision on the observation that the cardinality of human-computer relation was changing as shown in the following figure.

Human-Computer relation over time (from http://tuprints.ulb.tu-darmstadt.de/5184/)

However, we are not there, yet. Currently, I see people holding their smartphones in front of their face to initiate, e.g., voice queries starting by "OK Google" or similar. So, we are not using everyday objects that we encounter in our daily lives to interact with them but use a vision-centric device as our key. As a results screen sizes are still increasing during the past years.

Smartphone Screen Size Over Time | SpecOut

One of the drawbacks of this vision-centric key to smart spaces is that is is by no means hands-and eyes free. Google and Siri are continuously improving the voice capabilities of their personal agents but they still rely on manual interaction and force the users to look at the screens. It appears as if we forgot about the "invisible" attribute of Weiser's vision. Invisible meaning that we did not perceive it as a device. Today, the smartphone is still an explicit device.
One day, in the very near future, we will have to decide if this is the way to go. Do we want to reuse this thing everywhere, while we are walking the streets, in our cars,...?

Maybe, this also(!) motivates the wish for Android Auto and Apple CarPlay to have the car as another cradle for your smartphone.

Scenarios like those described in http://www.fastcodesign.com/3054733/the-new-story-of-computing-invisible-and-smarter-than-you are still far away. A video demonstrates their room E.

Prototypes like this are already existing in the labs and maybe, it is time to leave the labs.

Amazon Echo is, maybe, a first step in this direction. As a consequence, it became the best selling item above $100 http://www.theverge.com/2015/12/1/9826168/amazon-echo-fire-black-friday-sales

In contrast to the scenario in the video above, users do not need to speak up. It can be used for voice querying and controlling devices http://www.cnet.com/news/amazon-echo-and-alexa-the-most-impressive-new-technology-of-2015/ So, let's see how this one evolves with regard to Weiser's vision. Maybe, we see comparable approaches soon.

Nuance opens their NLU to developers

Nuance just opened their NLU platform as a beta to developers https://developer.nuance.com/public/index.php?task=mix. 

It is more than simply NLU but a full stack including speech recognition that can be used in own applications as shown in their promotion video.

Similar to efforts of NLU startups residing under .ai, Nuance Mix is able to detect an intent and user defined entities from entered sentences. The possibility to also employ Nuance ASR, however, makes it more complete than those efforts. Maybe, this has to be seen as an attempt to strenghten Nuance approach to a virtual assistant that they call Nina. Nina has been out for a while but did not receive much attention so far.

The market of virtual assistants is already somewhat populated. Google Now and Apple's Siri are well known and established. Others, like Microsoft's  Cortana also try to gain attraction. Recently, Microsoft opened Project Oxford as a cloud - based tool for the creation of smart (voice centric) applications. A comparable, but maybe more advanced offer is IBM Watson which is available for some time. Another one is Amazon Echo that also opened their platform to developers.

It appears that spoken language technology is mature enough to be really useful. Good news for developers who want to play around with voice interaction to control applications in the internet of things. Currently, there is a plethora of SDKs available that can be used for free. The current question is not if we will see more spoken interaction with everyday things in our lives, but who will win the race for a sufficient number of users and their data. Maybe, Nuance is already too late with Nuance Mix to enter that market. Maybe, they can step in nevertheless, relying on their year-long dominance in speech recognition

NLU vs Dialog Management

Recently, I stumbled across a blog from api.ai that their system now supports slot filling: https://api.ai/blog/2015/11/09/SlotFilling/. Note, that my goal is not on blaming their system.

Currently, I observe that efforts towards spoken interaction coming from cognitive computing are still not fully aware of what has been done in dialog management research in the past decades, and vice versa. Both parties are coming from different centers in the chain of spoken dialog systems.

While the AI community usually focuses on natural language understanding (linguistic analysis) the spoken dialog community focuses on the dialog manager as the central point in this chain.

Both have good reasons for their attitude and are able to deliver convincing results.

Cognitive computing sees the central point in the semantics which should also be grounded with previous utterances or external content. Speech input and output is in this view restricted to be some input into the system and some output. Dialog management can be really dumb in this case. Resulting user interfaces are currently more or less based on queries.

The dialog manager focused view regards the NLU to be some input into the system while the decision upon subsequent interaction is being handled in this component. Resulting user interfaces range from rigid state-based approaches over the information state update approach up to statistical motivated dialog managers like POMPD.

My hope is, that both communities start talking to each other to better incorporate convincing results of "the other component" to arrive at a convincing user experience.

Microsoft researchers expect human like capabilities of spoken systems in a few years

Researchers at Microsoft believe that we are only a few years away from equal capabilities of machines to understand spoken language as humans do. Although many advances have been made in the past years there are still too many challenges that need to be solved.This is especially true for distant speech recognition that we need to cope with in daily situations.  Maybe, their statement is still a bit too optimistic. However, as systems are available already and people start using it they are right in their assumption that these systems will make progress. We just have to make sure that voice based assistants like Cortana are used at all. Currently some of these systems seem to be more a gimmick to play with until users become bored of it. Hence, they are actually dammed to improve fast to also be helpful. 

http://news.microsoft.com/features/speak-hear-talk-the-long-quest-for-technology-that-understands-speech-as-well-as-a-human/

How to share a DocumentBuilder instance in Java

I tried to optimize parsing of XML document in JVoiceXML by using a DocumentBuilder as a static member:

    private final static DocumentBuilder BUILDER;

    static {
        final DocumentBuilderFactory factory =
                DocumentBuilderFactory.newInstance();
        factory.setNamespaceAware(true);
        factory.setIgnoringComments(true);
        DocumentBuilder builder = null;
        try {
            builder = factory.newDocumentBuilder();
        } catch (ParserConfigurationException e) {
            e.printStackTrace();
        }
        BUILDER = builder;
    }

When I used this construct by multiple threads, I stumbled across the following exception:
 
    org.xml.sax.SAXException: FWK005 parse may not be called while parsing

A quick search with google came up with results like this.Bad news: Parsing XML in Java is not thread safe. An immediate solution is synchronization. However, synchronization adds a lot of overhead and is actually only needed, if calls are actually coming from different threads. A better solution for me was to use ThreadLocal. My current soultion in JVoiceXML is the following:

    private static final ThreadLocal LOCAL_BUILDER
        = new ThreadLocal() {
        @Override
        protected DocumentBuilder initialValue() {
            final DocumentBuilderFactory factory =
                    DocumentBuilderFactory.newInstance();
            factory.setNamespaceAware(true);
            factory.setIgnoringComments(true);
            DocumentBuilder builder = null;
            try {
                builder = factory.newDocumentBuilder();
            } catch (ParserConfigurationException e) {
                e.printStackTrace();
            }
            return builder;
        };
    };

The implementation of ThreadLocal usage is very fast. However, creation of new objects is also very fast in current JVMs. I still have to check if caching really makes sense in this scenario, espacially, in the light of possible memory leaks as indicated by Nancy Deschenes.

JNI causes XML parsing problems when using a custom ClassLoader

Recently, I discovered some strange problems, when I was trying to parse XML files when being called via JNI. The special thing here was that this method was called from a class that I loaded with a custom Java ClassLoader. Here, I got a NullPointerException when I was trying to obtain an XPathFactory via

XPathFactory xpathFactory = XPathFactory.newInstance();

Usually, this works out of the box, but I got an exception similar to this one: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6245257 
The strange thing is: The same code works if it was not called from a JNI method but from my own Java method.

Searching via Google reveiled a preliminary solution at https://forums.oracle.com/forums/thread.jspa?threadID=1626887 
They suggest to use

ClassLoader cl = Thread.currentThread().getContextClassLoader();
Thread.currentThread().setContextClassLoader(new URLClassLoader(new URL[0], cl));
...
Thread.currentThread().setContextClassLoader(cl);

This code pretty much resets the ClassLoader to some more reasonable value. However, my problems did not vanish completely. The next error popped up, when I utilized some spring beans parser that threw strange exceptions about some XSD files claiming they were not readable. Again, the same code worked when being called from plain Java. So I started to dig into the problem. JNI guarantees that the ClassLoaders are the same when a Java method is called from JNI that were used to load the JNI library (see section 11.2 at http://192.9.162.55/docs/books/jni/html/design.html). This was true. A quick check with

System.out.println(getClass().getClassLoader());

showed the same ClassLoader for the time the library was loaded and for the time when the method from JNI was called. However, this was not true for the context ClassLoader. When the library was loaded, the method returned the usual URLClassLoader, but when being called from JNI, it simply returend null, which is the system ClassLoader. This was also true if the code above was used. This means that you will be able to access all class files that reside in rt.jar plus those that you added in your own class loader. Other classes that are subject to the endorsed capabilities will not be accessible anymore (see http://docs.oracle.com/javase/6/docs/technotes/guides/standards/). This includes the XML parsing libraries.

My solution for now was to adapt the context ClassLoader with

ClassLoader loader = getClass().getClassLoader();
Thread.currentThread().setContextClassLoader(loader);

This sets the context ClassLoader to the same value as it was when the JNI library was loaded.