RFC: Web Platform Installer (WebPI) with an offline feed mode for bandwidth savings

My wife is in the Kingdom of Swaziland this evening. If she was a developer over there and wanted to use Web Platform Installer to get Visual Studio or some large download she would have to patiently bring the bits down over 3G. If she was very well off, she might have ASDL with a maximum speed of 512kbps down, most likely with a cap. I've talked about limited connectivity in African countries and other nations (*cough* New Zealand *cough*) before on this blog and it's no fun. I've told the wife in the past that I'm happy to live overseas but I needs me some bandwidth. It sucks when tools you like don't conserve your bandwidth.

Making sure products are available and convenient for customers without a "fat pipe" is a passion of mine and I'm always the guy in the meetings at work to remind people within the Redmond Reality Distortion Field that not everyone in the world has 35 megabit fiber to their house.

We've also been trying to push for transparency and openness in all things (no more secrets, I like to say) and it usually comes down to this: They don't like to announce stuff if they aren't sure they are going to ship it. That's cool, and I get it.

But, when I saw that the folks on the WebPI team have heard you and I and told me they are working on an offline experience for WebPI today I had to share it with you, Dear Reader. Here's the deal, I'm looking at daily builds off the developer's machine. This IS NOT the final experience and this IS NOT anything other than a daily. They are looking at shipping this soon, but they told me to put lots of disclaimers. Disclaim! There it is. As this is a daily build, there are likely more questions than answers, so I'll show you how it works now, with my commentary throughout.

Web Platform Installer Offline

The problem is, of course, that the Web Platform Installer assumes that you and the 50 people at your company sharing internet are cool with downloading the same x gigs 50 times. It also assumes that your system IS connected to the internet in some way. It's a wasteful way for us to get you bits and everyone loses.

I think the ideal use case is that someone who has the bandwidth and the time downloads all the apps they want then puts them on a DVD or a network share and then all the consumers point to that offline cache. Companies could have a share or feed that's refreshed monthly, for example.

Products in WebPI have an "AppID" or Product ID. You'll need to know the AppID for the thing or things you want to install. For example, with the online Web Platform Installer if you want to install "all the web stuff" in one click, you'd use this URL http://www.microsoft.com/web/gallery/install.aspx?appid=VWDorVS2010SP1Pack.

See the "VWDorVS2010SP1Pack" part? That refers to an AppID inside of the Feed that Web Platform Installer consumes. Everything in the feed has an AppID and you can string them together. For example, you can say http://www.microsoft.com/web/gallery/install.aspx?appid=MVC to get ASP.NET MVC 3 for VS2010. You can add lots of AppIDs separated by semicolons ";" to get multiple products and all their dependencies.

I am testing a spike of an offline command line tool called wpiutil.exe but the plan is to merge it with the Web Platform Installer Command Line tool. So, even though I'm showing two tools, pretend there are one.

You can find an AppID inside the Atom Feed that the Web Platform Installer consumes. The main feed is at https://go.microsoft.com/?linkid=9767054 which isn't really discoverable, so you can use the command line tool to find what you want.

I can run WebpiCmdLine.exe /list:all and get a big list of products and applications, then collect the IDs I want to install or cache offline for later.

Current language of installers is English


--Previously Installed Products
ID                   Title
----------------------------------------
AppFabric            Windows Server AppFabric
AppFabricSetup       Windows Server AppFabric Setup
*SNIP*
MVC2                 ASP.NET MVC 2
MVC3                 ASP.NET MVC 3 (Visual Studio 2010)
MVC3Loc              ASP.NET MVC 3 Tools Update Language Packs
MVC3Runtime          ASP.NET MVC 3 Runtime
*SNIP*
--Applications
ID                   Title
----------------------------------------
AcquiaDrupal         Acquia Drupal
Atlas                Atlas
*SNIP*
Umbraco              Umbraco CMS
WordPress            WordPress

Scott Feedback: I told the team that I thought these AppIDs could be also on this page in the GUI. We could add "Product ID:" to screens like this?

So now I can do this:

C:\>wpiutil.exe /create-shadow /shadow-path:c:\webpicache /product-ids:MVC3;IISExpress

Loading products in webpi feeds ...

Creating shadow for following products:
MVC3
NETFramework4
WindowsImagingComponent
MVC3Installer
IISExpress

Processing feed: 'https://go.microsoft.com/?linkid=9767054'

Processing resources ...

Processing product entries ...
Processing product entry: 'MVC3Installer'.
Processing product entry: 'MVC3'.
Processing product entry: 'NETFramework4'.
Processing product entry: 'IISExpress'.
Processing product entry: 'WindowsImagingComponent'.

Done.

See how the tool got the the dependencies as well? Here's the folder structure that was created locally:

Now I can point the command line tool at my new custom feed. My custom feed contains ONLY what I've downloaded. It's a subset, although I suppose a company COULD make a complete (or at least a large) shadow copy of everything and keep it on a server.

C:\>WebpiCmdLine.exe /xml:"C:\webpicache\feeds\shadowfeeds\shadow-webproductlist.xml" /List:All

Successfully loaded metadata from feed C:\webpicache\feeds\shadowfeeds\shadow-webproductlist.xml

Current language of installers is English

--Available Products
ID                   Title
----------------------------------------
MVC3Installer        ASP.NET MVC 3 Tools Update Installer
IISExpress           IIS 7.5 Express
MVC3                 ASP.NET MVC 3 (Visual Studio 2010)
NETFramework4        Microsoft .NET Framework 4

Then I can install the products I want like this:

C:\>WebpiCmdLine.exe /xml:"C:\webpicache\feeds\shadowfeeds\shadow-webproductlist.xml" /Products:MVC3

Here's some of the my personal feedback to the team in no order:

• There should be as much visual feedback as possible from the command line tool, given that it'll be run over slow networks (even 3G) so something like a (345kb of 56 megs) updating text area...
• A way to switch the Web Platform GUI to be completely offline by pointing it to a local feed.
• The command line tool should be smart about relative vs. absolute paths, as well as UNC, or even running your company's custom internal Web PI server.
• Maybe consider using BITS (background internet transfer service) like Windows Update so folks could run it in the background over many days.
• "Shadow" isn't a good name as it sounds like the Shadow Copying feature in Windows. Just call it offline.
• More documentation on how customers can make custom feeds for their products. Just as NuGet gets libraries for Visual Studio 2010 projects, WebPI consumes feeds of MSIs and their dependencies.
• It must be smart about resuming after a dropped connection or call and must retry corrupt downloads.
• Group policy support so that IT Administrators could control how WebPI works inside a large enterprise and where it loads files from.
• Should have a smart default path like c:\webpicache or .\webpicache if it's run without a full path.

Now, this is your (and my) opportunity to provide comments and suggestions to the team and affect the final direction of the product or future versions.

From what I've shown you, Dear Reader, my questions to YOU are:

• Will this be a good feature?
• Will it help you?
• Is it useful or useless?
• Is it important to be a GUI or is a Command Line good too?
• What has been forgotten?
• What's a "must have" and what's a "don't need?"

Discuss, they are listening!