Introducing Lync 2010 Super Simple Auto Answer Video Kiosk with Full Screen

If you check out my blog archives or check out the Remote Work category you'll see that I'm always tinkering with my remote work situation. I'm the most interested in high quality and pervasive video. I'm so interested in this that for a while I was running a 10 hour a day persistent "Portal" between Portland and Seattle. I still highly recommend this for co-located teams that have the bandwidth to burn. It's great to be able to turn one's head and see their teammate right there by their side - even though they are 200 miles away.

I recently picked up a pair of Logitech BCC950 Conference Cams as a possible replacement for the very expensive "RoundTables" that some rooms at Microsoft have. The RoundTables are lovely but they are becoming scarce at the office and the Logitech is literally one-tenth the price. I'll blog a full and detailed review of the BCC950 later on this week but for now my biggest issue was that the Video Kiosk software I was using was starting to show its age. It's flaky and unreliable and build on the Office Communicator 2007 interfaces while I've been on Lync 2010 for a while.

Additionally, the researchers that wrote the software are always adding features I don't need for hardware I don't have. My remote buddy Damian Edwards and I decided we needed to make a software break.

Features

We want these simple features to start:

• Auto-answer of video calls - possibly with some whitelist for security
• Auto-fullscreen of video calls on the remote machine - the single purpose kiosk in my Seattle office
• Presence information and a simple UI for making calls - by simple I mean "my boss's boss" simple
• Remote control of Pan Tilt Zoom (PTZ) features on the same - ideally using the standard "inbox UVC" drivers and no 3rd party software

Tonight I sat down and did the first three of these and put it on GitHub. I call it - wait for it - the Lync 2012 Super Simple Auto Answer Video Kiosk with Full Screen since the name "SmartGlass" was already taken. ;)

I searched ALL over to find out if there was SOME sample code out there that would just auto-answer a call from Lync and start video. I could find dozens of samples that made calls, that started chats, but none that would answer and auto-start video. You'd think this would be the FIRST thing that folks would want to do. I can only assume it's not a setting for security reasons.

Auto-Answering Lync Calls with Video

Now, it's late and there's likely problems so pull requests are welcome if I have done something stupid. Lync is complex and while you'd think there'd be an "AutoAnswer = true" it's actually a more complex API than that. I started from this MSDN article on "Responding to a Lync Conversation Invitation."

var lync = LyncClient.GetClient();
var conversationmgr = lync.ConversationManager;
conversationmgr.ConversationAdded += (_, cmea) =>
{
    bool IncomingAV = false;
    StringBuilder sb = new StringBuilder();

    //Is this an audio/video invitation?
    if (cmea.Conversation.Modalities[ModalityTypes.AudioVideo].State == ModalityState.Notified)
    {
        if (lync.DeviceManager.ActiveAudioDevice != null)
        {
            sb.Append("Incoming call from ");
            IncomingAV = true;
        }
        else
        {
            cmea.Conversation.Modalities[ModalityTypes.AudioVideo].Reject(ModalityDisconnectReason.NotAcceptableHere);
        }
    }
    if (cmea.Conversation.Modalities[ModalityTypes.InstantMessage].State == ModalityState.Connected)
    {
        sb.Append("Incoming IM from ");
    }
    sb.Append(String.Join(", ", cmea.Conversation.Participants.Select(i => i.Contact.Uri)));
    Debug.WriteLine(sb.ToString());

    if (IncomingAV == true && Properties.Settings.Default.autoAnswer == true) //I added that setting later on
    {
        InitiateAVStream(cmea.Conversation);
    }
};

You watch for a Conversation to start and see if it's an Audio/Video on. If it is, then we call our InitiateAVStream method. You can't do all this stuff synchronously as Lync is full of async native COM APIs and events that you need to respond to. Here we accept the video which lets us see who called us but doesn't yet start OUR video. Remember "we" are the dumb Kiosk who is receiving a call from me.

private static void InitiateAVStream(Conversation pConversation)
{
    if (pConversation.State == ConversationState.Terminated) { return; }

    if (pConversation.Modalities[ModalityTypes.AudioVideo].CanInvoke(ModalityAction.Connect))
    {
        var video = (AVModality)pConversation.Modalities[ModalityTypes.AudioVideo];
        video.Accept();

        //Get ready to be connected, then WE can start OUR video
        video.ModalityStateChanged += _AVModality_ModalityStateChanged;
    }
}

See how they are chaining handlers? I think this code could be made cleaner with a series of nested closures like above in the ConversationAdded example, but then again, maybe not. It'll get four deep before we're done.

Now the call is being connected but perhaps not yet. When its state changes the VideoChannel has opened up and we watch for the video to be received.

static void _AVModality_ModalityStateChanged(object sender, ModalityStateChangedEventArgs e)
{
    VideoChannel vc = null;
    switch (e.NewState)
    {
        //we can't start video until it's connected
        case ModalityState.Connected:
            if (vc == null)
            {
                vc = ((AVModality)sender).VideoChannel;
                vc.StateChanged += new EventHandler<ChannelStateChangedEventArgs>(_VideoChannel_StateChanged);
            }
            break;
    }
}

As the video starts up, I can see if the system is ready for the Kiosk to start its video. If so, we call BeginStart (and the SDK says we HAVE to call EndStart, so watch out!).

static void _VideoChannel_StateChanged(object sender, ChannelStateChangedEventArgs e)
{
    VideoChannel vc = (VideoChannel)sender;

    //Are we receiving? Let's try to send!
    if (e.NewState == ChannelState.Receive)
    {
        if (vc.CanInvoke(ChannelAction.Start))
        {
            vc.BeginStart(videoCallBack, vc);
        }
        else {  Debug.WriteLine("CanInvoke said NO!"); }

        //Go looking around for the IM Window (there had better just be the one we just started)
        // and force it to the foreground
        IntPtr childHandle = UnsafeNativeMethods.FindWindowEx(IntPtr.Zero, IntPtr.Zero, "IMWindowClass", null);
        UnsafeNativeMethods.SetForegroundWindow(childHandle);

        //Try to get the video to go full screen by pressing F5
        WindowsInput.InputSimulator.SimulateKeyPress(WindowsInput.VirtualKeyCode.F5);
    }
}


private static void videoCallBack(IAsyncResult ar)
{
    ((VideoChannel)ar.AsyncState).EndStart(ar);
}

I'm pretty frustrated as while this is SUPER powerful, it's also SUPER complex for basic scenarios in my opinion. I think there's an opportunity here for a small layer on top of Lync that handles the 80% cases like the small Lync Abstraction  layer introduced in this "ScreenPop" example application.

The Goodness - The WPF Controls

At this point in the code I had everything working in a Console Application. You can go cherry-pick that commit if you want just a Console app that Auto-Answers video calls from Lync.

Even though I NEED to stop as I've got it working in a Console and I should be sleeping I noticed this in Visual Studio and it was too epic to not try.

You know how it is. It's 2am, you're done with your goals. OF COURSE you're going to try to convert your Console App to a GUI before bed. Of course.

Turns out there's a MESS of visual controls that you can put into existing applications to make them Lync-ified in literally minutes.

Ok, awesome. I took the basic UI and added a checkbox for "Auto Answer."

(ASIDE: You DO realize that the outline of the "unknown face" in Lync there looks an AWFUL lot like Bill Gates' legendary 1977 mug shot, right? I just noticed that.)

Anyway, then I made two settings, one for my "sip" address (that's in my app.config file as "sipEmailAddress" and one boolean for AutoAnswer. The complete XAML is just:

<Window x:Class="SuperSimpleLyncKiosk.Main"
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        xmlns:properties="clr-namespace:SuperSimpleLyncKiosk.Properties"
    xmlns:controls="clr-namespace:Microsoft.Lync.Controls;assembly=Microsoft.Lync.Controls"
    Title="The World's Simplest Lync Kiosk (with Auto Answer, too!)" Height="Auto" Width="Auto">
    <Grid RenderTransformOrigin="0.5,0.5">
        <Grid.RenderTransform>
            <ScaleTransform ScaleX="3" ScaleY="3"/>
        </Grid.RenderTransform>

        <StackPanel Orientation="Vertical" HorizontalAlignment="Center" VerticalAlignment="Center">
            <StackPanel Orientation="Horizontal" HorizontalAlignment="Center" VerticalAlignment="Center">
                <!-- Show the presence indicator. Hover over the icon to see the contact card.
                 Set Source to a valid SIP URI in your organization.  -->
                <controls:PresenceIndicator 
                x:Name="Presence"  
                Source="{Binding Source={x:Static properties:Settings.Default}, Path=sipEmailAddress, Mode=OneWay}" 
                PhotoDisplayMode="Large" 
                />
                <!-- Use the DisplayName property from PresenceIndicator to show the user's name -->
                <TextBlock 
                Text="{Binding DisplayName, ElementName=Presence}" 
                Margin="4,0,0,0" 
                VerticalAlignment="Center"
                />
                <controls:StartVideoCallButton Source="{Binding Source={x:Static properties:Settings.Default}, Path=sipEmailAddress, Mode=OneWay}" x:Name="startVideoCall" />
                <controls:ShareDesktopButton Source="{Binding Source={x:Static properties:Settings.Default}, Path=sipEmailAddress, Mode=OneWay}"/>
            </StackPanel>
            <CheckBox  IsChecked="{Binding Source={x:Static properties:Settings.Default}, Path=autoAnswer, Mode=TwoWay}" Content="Auto Answer Video Calls"/>
        </StackPanel>
    </Grid>

</Window>

I also added a 3x transform to scale ALL these default controls so they'd look good on the 42" TV that is sitting in my office. Because they are native WPF vector controls they just scale perfectly to high resolutions without raster jaggies.

Then I added a call to make the app Maximized by default:

this.WindowState = System.Windows.WindowState.Maximized;

And it looks like this when running:

And when I call it automatically answers. Looks like everyone's asleep and they've turned the lights out!

Ah, I but I wish it was full screen so the people in Redmond don't need to do anything or touch anything...

The Badness

Some nights you're programming and you say "may God have mercy on my soul for this line of code."

— Scott Hanselman (@shanselman) July 3, 2012

I can auto-answer calls but sometimes the window isn't in front and once it gets in front there's no programmatic way to tell Lync to go Fullscreen with video.

Two bad problems there. Both solved by breaking all the rules. I get the Window Class with a big assumption that the Kiosk only as one chat window open and then I "push" F5 which is the Lync hotkey for fullscreen video.

//Go looking around for the IM Window (there had better just be the one we just started)
// and force it to the foreground
IntPtr childHandle = UnsafeNativeMethods.FindWindowEx(IntPtr.Zero, IntPtr.Zero, "IMWindowClass", null);
UnsafeNativeMethods.SetForegroundWindow(childHandle);
//Try to get the video to go full screen by pressing F5
WindowsInput.InputSimulator.SimulateKeyPress(WindowsInput.VirtualKeyCode.F5);

Those last two, of course, are calls directly into Win32 from .NET:

public static class UnsafeNativeMethods
{
    [DllImport("user32.dll")]
    public static extern IntPtr FindWindowEx(IntPtr parentHandle, IntPtr childAfter, string className, string windowTitle);

    [DllImport("user32.dll")]
    public static extern bool SetForegroundWindow(IntPtr hWnd);
}

But, it works! It's scandalous that this isn't built into the Lync SDK. Developers who are fans of Lync or who work on it all the time will say that my attempt at a "poor man's Kiosk" is silly and that I really want to use "UI Suppression in Lync" and just make an app that hosts Lync rather than automates Lync. They are likely right. However, frankly, it looked super-hard and I was tired. So, ya. If anyone wants to work on the Kiosk with me to make it simple answer and start video and do it all without showing Lync, that'd be awesome.

Thanks

I also want to make a special nod to the InputSimulator library. It's amazing and it just works. It's WAY WAY better than SendKeys which you should stop using NOW.

The Windows Input Simulator provides a simple .NET (C#) interface to simulate Keyboard or Mouse input using the Win32 SendInput method. All of the Interop is done for you and there's a simple programming model for sending multiple keystrokes.

Enjoy!

Lync Developer Resources

• CodeLync
• Developing Lync (Tom Morgan)
• Justin Morris on UC
• Lync Development by Michael Greenlee
• Lync'd Up (Tom Arbuthnot)
• The Modality Systems blog

Related Links

• Get involved in Open Source today - How to contribute a patch to a GitHub hosted Open Source project
• Review: Living, working and using the Cisco Umi personal telepresence system. All that and bag of chips?
• Hanselminutes Podcast 242 - The Plight of the Remote Worker with Pete Brown
• The Weekly Source Code 57 -Controlling an Eagletron TrackerPod with C# 4, ASP.NET MVC and jQuery
• 30 Tips for Successful Communication as a Remote Worker
• DIY: Making a Very Wide Angle Webcam on the Cheap
• Building an Embodied Social Proxy or Crazy Webcam Remote Cart Thing
• How to Collaborate with Remote Employees with Office Communicator 2007 R2
• Virtual Camaraderie - A Persistent Video "Portal" for the Remote Worker
• Working Remotely from Home, Telepresence and Video Conferencing: One Year Later
• Microsoft - Surviving First Three Weeks as a Remote Employee