Category Archives: Uncategorized

An Intro to “TestOps”?

A few of my co-workers and I were just talking about testing trends and I was recently directed to Seth Eliot’s blog (here) by another colleague. Seth is currently at Amazon and he was at Microsoft for approximately six years in various test management and test lead roles. His work experience is noteworthy because he’s been working for very large organizations that deliver software at a rapid pace. He’s put into words the thoughts I’ve been muddling around on for the last five years.

It’s an interesting time for QA individuals. Due to several factors, their world might be a lot different than it has been in previous years. This is the first post in a short series of blog posts on the topic of testing trends. This first post will just lay some groundwork. Subsequent posts will go deeper and offer up possible solutions.

I used to work on a development team at a large organization that would take six to eight weeks to document and refine 75-pages of requirements. They’d then hand these over to the development teams who would write code for three to six months. Although the developers would do some unit testing and even some functional testing in a dev environment, they would depend on a sizable QA team to validate that the build was meeting expectations. This QA effort would take another two to four months. Altogether, that’s between seven and ten months to get this project out the door and into users’ hands.

In this world, QA had weeks or months to write manual test cases and execute them, but it was rarely enough time to test the new functionality and fully regression test the application.

Enter Agile!

The popularity of agile has definitely surpassed critical mass! Enterprise development teams NOT incorporating several agile practices are now the exception. This often means development teams are reducing their release cycles from months (years?!) down to four weeks or even days. This provides tremendous business value as stakeholders get to see their most-valued requirements implemented and working in a dev environment faster than ever.

Unfortunately, the pervasiveness of agile has been limited to those writing code and not to the greater set of people involved in the software development process. Shortening your release cycle to two weeks doesn’t do you much good if QA requires weeks/months to run their tests. The bottleneck has now shifted from developers over to QA to get what developers are showing to stakeholders out the door and into the hands of end-users! (This same pressure applies to Ops, but that’s a different blog series.)

I’ll dive into these topics in future posts, but let me give you a teaser on some of my thoughts…

  • Move QA earlier in the cycle
  • Reduce your dependence on manual tests
  • Increase your dependence on unit tests
  • Ask yourself how your automated functional testing is going
  • Learn more from production

Would you like to discus these topics in person? Come join us at the St. Louis Silver Linings Conference on Oct 30. Admission is just $30 for a whole day of breakout sessions on DevOps, the cloud, and more!

Monitoring, Measuring, and Learning for Continuous Improvement

It is a very exciting time to be in technology! I think most of us are attracted to this industry because we love to play with cool new things. This technology could range from entertaining gadgets/hardware to new software, but this also means we have to be adaptive. Those who can’t adapt may be able to survive, but those that can will thrive.

When you think about adapting in terms of application development and operations, the high-level process is pretty intuitive. Monitoring and monitoring are just the acts of capturing raw bits of information about your application and the environment it resides in. The analysis and learning are when you take that raw data and transform it into something that can be used to improve an aspect of your application.

When you implement changes based on what you learned from your monitors, you can expect to see payback based on one or more of the following:

  • A reduction in mean time to detect problems
  • A reduction in mean time to resolve (MTtR) problems
  • Improved success in achieving business goals

Azure Management Portal

Your virtual machine DASHBOARD and MONITOR tabs will show you the CPU usage, disk read bytes/second, disk write bytes/sec, incoming network traffic, and outgoing network traffic. Your Azure Web Sites provide slightly different, but very similar metrics and the screenshot of my application’s metrics are below.

You can also set up email alerts using the Azure Management Portal based on a metric that exceeds a selected threshold. This is done through the Management Services option in the navigation pane of the portal.

The above dashboard metrics don’t require any additional work on your part – they’re on by default. In addition, on the Configure tab of a Web Site or Virtual Machine, you can set up Endpoint Monitoring. This allows you to test one or more URLs from any of our Azure data centers across the planet. Azure Web Sites can only use Endpoint Monitoring if they are in either Basic or Standard mode.

Application Insights

Application Insights is a new service under the Visual Studio Online umbrella of services and it provides all kinds of rich monitoring features. It’s currently in public preview, so anyone can check it out. While you don’t need to store code in Visual Studio Online to use the service, but you do need a Visual Studio Online account.

Configure an application

If your application is already deployed on a server, you only need to do the following to get started:

  1. Download ApplicationInsights.config from your Application Insights portal. This configuration file contains everything you need to connect your application to the Application Insights service. You can modify some additional settings as well.

  1. Download and install Microsoft Monitoring Agent on the server. Right now, there’s a pre-release version of the agent that you should download from the Application Insights portal. Eventually, the same agent can be used to send data to Application Insights or System Center. (If you already have the System Center agent installed, you’ll have to uninstall it to utilize this new agent.)

Special note: We are working to remove the dependency on the agent as much as possible, so that you don’t need that separate install. Ideally, the necessary telemetry components just ride along with your application.


If you are creating or editing an application in Visual Studio, you can follow these steps to connect to Application Insights:

  1. If you have Visual Studio 2013, you can download and install Application Insights Tools for Visual Studio from here. Older versions of Visual Studio will lack some of the new integration, but you can just add the desired Application Insights components from NuGet.
  2. If you have an existing Visual Studio project, you can then right-click the project and select Add Application Insights Telemetry to Project…
    1. Confirm the selected Visual Studio Online account
    2. Click Configure settings to change the name of the container you want to upload telemetry to.
  3. Developers can also opt-in to Application Insights right when they create a Visual Studio project.
  4. In any case, you should still consider downloading the Microsoft Monitoring Agent to get the full set of telemetry.

Special note: For Azure PaaS web and worker roles, you have to modify cscfg and add startup task. Azure Web Sites are not yet supported with Application Insights.

What can we monitor/measure?

Application Insights currently supports the following application types: .NET Web Applications (sites or services), Java web applications, Windows Store and Windows Phone Apps. Supported platform list currently includes Tomcat 5, 6, or 7, JBoss 6, Windows Server 2008 or 2012, and Linux (CentOS, Red Hat, Ubuntu).

From the OVERVIEW tab, you can see a list your monitored applications and some average stats. Each of these applications is really a container where you send data. For example, you may have the same application installed on several servers in a web farm, but you want to treat the farm as one application. You can view a list of monitored servers which also displays agent version, OS version, and the last reported “heartbeat”.

Alerts was very recently redesigned to provide a more comprehensive experience. You can set up alerts for availability or specific metric threshold violations. The Alerts page displays a rich alert history.

You can set up dashboards with the metrics you care about most. You can even have multiple dashboards focusing on different metrics. Created dashboards can be private or shared with others.

How do we use this to learn and improve?


The easiest thing to set up is availability tests. There is no code required and you can test a web application built on any platform.

To set this up, you need to create a synthetic monitor – either a Single URL or a Multistep Web Test. In some cases, directly hitting a URL with the Single URL synthetic test is all you need. A good example would be checking on your home page. In other cases, verifying availability can be a bit more complex. Let’s say I don’t consider my application available unless I’m confident users can find and add items to their virtual cart, and maybe even place the order. In order to do this, I need a sequence of steps, so the Multistep Web Test allows me to upload Visual Studio Web Performance Tests.

These are called “synthetic” because they are not activity from real users. This is important to understand. Your actual user activity could still be failing, but we’ll come back to that in just a bit.

Tracking availability is done by using one of our 16 Azure data centers scattered across the planet. By running the tests from these various locations, you can get a sense of how responsive your app is to users near those locations.

Each green dot or red x indicates the result of a synthetic monitor. You can click into a failed multistep test to see the individual steps and even download the test result into Visual Studio for further analysis by developers if necessary.

One of my favorite features is the subtle dots just below the timeline. Whenever you see these, they indicate a change that could explain why your metrics suddenly differ. Blue dots indicate a deployment and green dots indicate a change to how you have configured your monitors. To get this information, you should use TFS to publish build information as part of your automated build.


There are three usage-related sections: Features, Users, and Environment.

The Features and Users sections track who and how users are consuming your application. We can track the active users/sessions/activities per day and we can track how long before known users return to the application.

If you are using the Visual Studio extension, you’re web site’s pages are already being tracked. You can insert the necessary JavaScript yourself as well by pulling it from the Application Insights portal. You can either paste the JavaScript into select pages or just drop it into your master page(s).

For many apps, you should consider logging authenticated users. If you do not, the service can’t know that the same user has just logged in using different browsers or devices and, therefore, will be counted as a different user in each case.

appInsights.start(“your app’s component ID”);

appInsights.appUserId = someUserIdentifier;

appInsights.accountId = someUserAccount;


If you want to log your own custom events from the user’s browser, add the following code. You’ll see these events on the Usage | Features | Events page. Note the option to create an event hierarchy so you can aggregate values. In this example, I’ll be able to see what items users are adding to my cart, but I could also just look at any cart activity.

if (appInsights) {

appInsights.logEvent(“cart/add”, // a hierarchical event of your choosing

{ Item : “notebook”, DiscountCode : “0” }, // any number of string key/value pairs

{ Quantity : 3 } ); // only allowed one numeric measure


In addition to the above usage data, we can also track a bit about the user’s environment. Which operating systems or browsers are users using? Their screen resolution may give you hints about the size or quality of the devices. Where are your users located in the world and what language are they using to hit your application?


In order to collect performance data, you need to have installed and configured the Microsoft Monitoring Agent.

Obviously, we’ll give you the metrics for CPU, memory usage, and network I/O. We’ll also provide the ever-popular number of exceptions thrown per second! (YUCK!)

When we are monitoring performance of a web application, two particular metrics are critical: request rate and response time. We’ll give you a list of the top 10 slowest requests by issue count. There’s a chart showing response time distribution, so you can quickly see if you have a lot of latency issues, but my favorite chart is the Response Time and Load vs Dependencies. On this graph, not only do we see request rate and response time overlaid, but there are various bars indicating the time spent awaiting calls to our application’s dependencies. These dependencies are automatically discovered by the Microsoft Monitoring Agent and may include calls to my application’s databases or to third-party web services. When you see those long response times at certain times of the day and you’ve been hard-pressed they were due to calls your stock quote service, now you’ll have some evidence. J

By default, if your application exceeds 90% of the maximum memory allowed within IIS or more than 50% of the machine’s physical memory, a memory event will be raised. If you open the details of this event, you can download a pair of memory dumps for analysis within Visual Studio 2013 Ultimate. One of the memory dumps is taken at the time of the problem and the other is a bit older, so you can compare the two.


The last major tab in Application Insights is the DIAGNOSTICS tab. And while I hope you never have to go there, I figured I should tell you a little about it. It is itself divided into four sub-tabs: Metrics, Events, Streaming Data, and Search.

The Metrics section allows you to overlay any number of performance counters. Select the desired metrics to appear on your chart and configure any alerts for any of these counters.

Events are recorded when metrics exceed a particular threshold or an exception is identified. You can double-click to open an event’s details. You may want to dive into the exception data or the call stack. If it’s a sneaky little exception, you can even download an IntelliTrace file so you can step through the application’s code as if you were running the Visual Studio debugger on the remote machine back when the event occurred!

Streaming Data is just used when running your application from Visual Studio. We can post events to Application Insights from Visual Studio that may assist you while you are developing/debugging/troubleshooting.

If you are using logging to record events, the Search tab can be used to search through your logs across all of your application’s servers. There’s some keywords and special punctuation to improve your search results. Check out this article to learn more about searching diagnostic logs.

I’ve found it helpful to use the following PowerShell scripts to control the monitoring of my applications. There are a small, but important set of parameters for these commands and you can read a bit more about them here.


Start-WebApplicationMonitoring –Cloud “Default Web Site” –OutputPath outputfolder –Mode Trace

Checkpoint-WebApplicationMonitoring // saves a log file

Stop-WebApplicationMonitoring –Local -All


This post was intended to make you aware of a couple of means of monitoring your application: built-in Azure monitoring and the more comprehensive Application Insights. It can often be difficult to know what data to log and pay attention to, but we hope that these options make that a little bit easier and less stressful.

One final note: Monitoring isn’t limited to production, but it should also be used in dev and QA environments. You may use slightly different settings, but you should be identifying problems before they get all the way to production. It also helps to get comfortable with these metrics while you are in development and not just at the stressful time of a production issue!

Additional Resources

Application Insights for Visual Studio Online

TechEd 2014 See How Microsoft Game Studios Uses Visual Studio Online Application Insights to Gain Early Warning of Performance Issues, Failures, and Pinpoint Root Causes

Set up your release to find and debug deployment problems

Publishing WCF Data Services to Azure Web Site

I’ve been developing a WCF Data Service and it was working great on my local machine. (How many times have you said that?) With the trickiest parts of the service in place, I thought I’d push it up to an Azure web site and get it working there. Unfortunately, I was getting the following error after publishing my service.

After troubleshooting for about an hour, I finally found a really helpful blog post describing the exact problem here. It appears the primary issue is that the .svc file contains a hard-coded reference to version of Microsoft.Data.Services. Like any good blog post, it also contains a couple of solutions. I simply removed the reference to any version number and it worked like a charm – both locally and in Azure!

Now, I’m gonna push my Code-First Migrations to SQL Azure. Wish me luck!

What’s New in Visual Studio Since the Launch?

Like many organizations in its size category, Microsoft is often perceived as slow-moving. However, that is certainly not the case with the Visual Studio team!

The Developer Division at Microsoft is made up of several teams, but most of them have been using agile techniques for many years. More important to customers, this team is now releasing in a more agile fashion as well. Below, I’ve listed all of the updates that have been rolled out since the launch of Visual Studio 2012.

It’s a pretty impressive list. Each of these releases is not just a set of bug fixes, but they also include new features. I’ve highlighted the cloud TFS releases in green. If you are not using this service, you may think you shouldn’t care about these items. The truth is that new TFS features are rolled out to the cloud service before they are bundled into the less-frequent updates for the on-premises TFS. This gives you a chance to see what’s coming. (You can sign up for a cloud TFS account here.)

Sep 12 – Visual Studio 2012 Launch (read more here)

Sep 13 – TFS 2012 Power Tools Launch (read more here)

Sep 17 – Team Foundation Service Sprint 36 (read more here)

Oct 08 – Team Foundation Service Sprint 37 (read more here)

Oct 29 – Team Foundation Service Sprint 38 (read more here)

Oct 31 – Team Foundation Service RTM (read more here)

Nov 02 – Git-TF 2.0 (read more here)

Nov 19 – Team Foundation Service Sprint 39 (read more here)

Nov 26 – Update 1 Released (read more here)

Nov 29 – New TFS 2012 Power Tools for Update 1 (read more here)

Dec 10 – Team Foundation Service Sprint 40 (read more here)

Jan 07 – Team Foundation Service Sprint 41 (read more here)

Jan 21 – Team Foundation Service Sprint 42 (read more here)

Jan 25 – OData for Team Foundation Service (read more here)

Feb 11 – Team Foundation Service Sprint 43 (read more here)

Mar 4 – Team Foundation Service Sprint 44 (read more here)

Mar 4 – Update 2 CTP 4 Released (read more here)

Mar 22 – Team Foundation Service Sprint 45 (read more here)

Last, but not least, a lot of the features in the above list are selected based on your votes in User Voice!

MS Holiday Store

I can’t wait to buy a Microsoft Surface, but where do I get one on launch date (Oct. 26th)? Well, if you live in a city that already has a Microsoft retail store, then you can go there. Several other cities (e.g. St. Louis) will be getting a holiday store.

Find a store near you here.

Windows Store is open!

I’m a little late reporting this, but I’m very excited. I’ve got two of my own Win 8 apps partially completed. Thanks to a design review from Clark Sell, I’ve got a few more changes to make that will add some great value! (Check out his other site here – not here.)

Windows Store app submission is open to all registered Windows Store developers – individuals and companies. Join the excitement and become part of the vibrant community of app and game developers for Windows. Create your Store developer account and submit your apps and games today!

So, where do I start?! Right here:

The New Nokia Flagship

The new Nokia phones powered by Microsoft Windows Phone 8 have been announced. While some seem to be less than excited about it, I think it looks great! The only thing I don’t like is the lack of a release date. Sad smile

Well, what does it look like?

I have the HTC Arrive (Sprint) and I really like the slide-and-tilt keyboard, but it looks like the trend across the industry is touch keyboards. If that’s all I’m going to get, I guess a bigger screen is a good thing. The screen on the 920 is 4.5” and has a resolution of 1280×768. (It wasn’t that long ago that I had to develop web applications on the assumption that a user’s desktop monitor had an 800×600 resolution!)


In my experience with the HTC Evo, I realized all too late that a big screen came with a battery that rarely lasted a day at work. This one claims to get 400 hours of 3G standby time, 10 hours of 3G talk time, 67 hours of music, or 6 hours of video. Six hours of video sounds awesome! I may be able to safely use this as I hop from city to city in our upcoming Visual Studio 2012 launch events.

It comes in four colors: yellow, red, white, and black.

Appearances aside…

It has 32GB of storage, but its unclear to me if it will support additional storage via MicroSD cards.

Like many of the up-and-coming Windows phones, it has a dual-core 1.5 Ghz processors. I’m not sure why I would need a dual-core processor in my phone and I would gladly sacrifice it in exchange for battery life if that was possible.

This new phone will also support Near Field Communication. I’m probably being a bit naïve, but I think this is the coolest thing. (I long for the day I can get rid of my wallet completely! I’d prefer a chip under my skin, but I guess it’s baby steps.)

It will have the ability to charge wirelessly. You just set it down on the included charging station. Check out the JBL PowerUp Wireless Charging Speaker for Nokia where you can listen to your phone’s music while it charges.

This one is listed at 8.7 megapixels for the main camera with 4x digital zoom. It has a front-facing camera. It also advertises a PureView camera phone intended to be “blur-free”. I don’t get excited by cameras and I’ll probably never understand what a megapixel really is.

And software?

It comes with the same great Microsoft software I use every day, but it now has Nokia Maps, Nokia Music Store, and Nokia Mix Radio. I haven’t looked into these last two, but I assume they are separate from my Zune/Xbox Music Pass and so I probably won’t need them.

I hear great things about Nokia Maps and the Nokia City Lens augmented reality feature looks like it might be useful as well.


Learn more at the official site here.

CHALLENGE: Enter a comment below defining the word ‘megapixel’ in simple, but correct teams.

They Listened…

It may not be the most prompt response, but you can’t say the Visual Studio team doesn’t listen to their customers!

On Sept. 6th, 2012, Matthew Johnson and Andrew Sarratori released a new Visual Studio Extension called the Visual Studio 2012 Color Theme Editor. You can find it here. Great work, guys!

It has some prebuilt themes and allows you to modify the fonts and colors of Visual Studio to your liking. I’ve included a few screenshots below so you can check it out.


I like a readable UI, but some developers feel REALLY strongly about the colors and contrast. Have fun!!

Team Foundation Server 11 Beta Power Tools

I’m constantly reminded that no one knows about the existence of so many of our Power Tools. Since Team Foundation Server is one of the developer tools I help demonstrate, I wanted to start by pointing out the helpful features that come with the Team Foundation Server 11 Beta Power Tools. (You can expect these to be updated and renamed the Team Foundation Server 2012 Power Tools very soon.)

Command-line interface – Additional command-line options using tf.exe.

Visual Studio Integration – Allows integration into the world’s best IDE! This provides some additional enhancements to Team Explorer to let you search for files in source control, clone build definitions, and integrate with Lync.

Check-in Policy Pack – A small number of commonly requested check-in policies

Process Editor – A GUI tool that alleviates some of the tedium of editing the process template and work item definition XML files

Best Practices Analyzer – Used to verify a Team Foundation Server deployment adheres to best practices. (NOTE: This requires PowerShell and .NET 2.0.)

Windows Shell Extension (x64) – Enables you to execute common source control commands from Windows Explorer. (NOTE: This requires .NET 4.0.)

PowerShell Cmdlets – Installs some useful TFS commandlets.

Test Attachment Cleaner – Utility to assist in purging old or irrelevant test data which can take up quite a bit of room on your database.

So, where can I get them? You can get them from Visual Studio’s extensions library or you can download them from here.

STLDODN 2012 Session Agenda Posted

We’ve posted the session agenda for St. Louis Days of .NET 2012!

IMPORTANT: This session agenda is subject to change up until the last minute – usually this is because speakers are finalizing their travel schedules.

We have fewer than 50 attendee tickets remaining (although there are still plenty of pre-compiler workshop slots available).