A Tour of Microsoft's Mac Lab

Today I'll stop my 3rd person perspective. I'm going to write a little bit more about what I do to help MacBU ship great software and provide some color around what's it's like to work on Mac software at Microsoft. Often when we have press events or special visits from our MVPs I'll give them a tour of the Mac Lab and explain what we do. They've always found it very interesting and so I thought I'd share a virtual tour of our Mac Lab. Let's get started:

Lab Door

We start with the door. The Mac Lab is about 2000 square feet of solid computers. The calendars you see down the side of the wall we use to mark team birthdays as well as special events. The CD on the door is an old Apple Software Restore CD from the last beige Mac Apple produced, the Power Macintosh G3.

The "Sandbox" and Plasma display

The first area in the Mac Lab is what we call the Sandbox. This is where we keep all significant hardware configurations Apple has released that run our products. We'll use the Plasma display to, watch DVDs and play games, uh er, I mean, do important training presentations. ;-) It's actually very useful because everyone can be in front of a computer and still see the main screen and follow along. Often other groups at Microsoft (the games group, hardware drivers group and even the Windows media group) will come and schedule time in the Mac Lab to test their software on the different hardware configurations.

Old iMacs and iBooks in the distance

More Macs

We have lots of Apple hardware. You can see here the old colorful iMacs along with some of the old iBooks. You can also see two of our Lab Technicians working on the backup systems, but more about that later. Up until a few months ago we had every significant hardware revision Apple has ever released since the dawn of time. We even had a section of the Lab we affectionately called the "Mac Nursery" where we kept all the older Macs going. We even had an old SE/30 and IIci and super expensive Mac II all connected via PhoneNet running Spectre, just for fun. It's always super fun to boot Word 1.0 or Excel 1.0 on these old machines and see how much things have changed. Due to lack of space in the Lab we had to put all of these older machines into storage and recycled the following:

• Macintosh (original)
• Macintosh SE
• Macintosh SE/30
• Macintosh Classic
• Macintosh Centris 610 <-- I had this one in my room during high school
• Macintosh IIci
• Macintosh IIsi
• Power Macintosh 7100/66
• Power Macintosh 7100/80
• Power Macintosh 7500/100
• Quadra 650
• Power Macintosh G3
• Duo Dock with Powerbook Duo 2300c
• Power Computing PowerCenter Pro 210

May they rest in peace.

ADIC Backup Robots

One of the realities of working with computers is that things fail. More often then you might think. We've used different backup robots, but ever since we moved to ADIC we've never had a robot failure. They just make great stuff. We have 3 robots and use Veritas Backup Exec. It works pretty well, as you can see:

Lots of backup tapes

We also do offsite backups just incase "The Big One" hits, but for regular use these tapes work just great, except when they don't. Recently we had a failure and lost 400 GB of data! We restored it from the tapes and then discovered that the Mac version of the Backup Exec agent had a corruption bug causing the restores to be compromised! Veritas folk were super responsive and they should have a new Mac agent out soon. Backup software and file systems are in the class of software that simply must work, all the time. Alas, this is not always the case.

I'm going to skip the "Build Lab" section of our Lab since it's very much in transition. Maybe I'll post about that later. For now, on to our automation system!

Mac Office is one of those "software in the large" projects. There's really no way a team of our size would be able to adequately test all of Office without the use of automated testing. Every day we get a new build of Office from the build machines, we copy it to our Xserve RAID connected to our dual G5 Xserve for access by our 249 automation machines. We then run thousands and thousands of tests on the new build. Typically we get 4 builds of Office each day: English Ship, English Debug, Japanese Ship and Japanese Debug. We run our entire battery of tests against all the builds and then report any failures to testers via email. The testers investigate the failures, log any bugs and then move on to their other duties as testers. This turns out to be very effective, if used properly, and over time it allows testers to focus on things humans do best, while letting computers verify the repetitious and mundane, but necessary, testing. It all started with our Blue and White G3s years ago. At first when testers would upgrade their test machines, instead of recycling the machines, "The Lab" would get them to add them to our automation machine pool. I think we had about 20 machines to begin with.

Venerable Old Bluies

After some time we started getting Gray G4s...

Speedy G4s

Then we upgraded to some dual proc machines...

Dual G5s

Then Apple give us a special gift. :-) You'd be probably be very surprised at the cost of running all these machines. There's the obvious electricity costs, but also cooling costs and even the physical space costs. Additionally, our system scales, not with CPU horsepower, but with quantity of machines. Most of the tests we run don't run significantly faster on a dual G5 vs. a single G4. So when Apple announced the Mac mini it wasn't minutes before we were considering how to use it for our automation system. The Mac mini has all the perfect qualifications:

1. Low power
2. Low heat
3. Small
4. Easy to pack together
5. Inexpensive

So we got a few to test things out...

Mixed Automation Rack

And then we made the big purchase:

Rack A, B and C = 150 Mac minis!

These work extraordinarily well. You might wonder how we control all these Macs. We use two methods: KVM switch box and Apple Remote Desktop. Thanks to our Lab Manager's great relationship with the IOGear folk we have a very reliable solution these days. It seemed like it took for ever to find a USB KVM switch box that didn't leave the machines "headless" after random reboots. The 8 port USB KVM from IOGear has been rock solid. So what does it look like to sit in front of 64 Mac minis? Like this:

64 Mac minis: 1 keyboard, monitor and mouse

This works very well when you must access the machines physically. Even so, just scanning each Mac for 1 second gets very old, very fast and Apple Remote Desktop comes to the rescue! When we need to see all the machines at once we just select them and BOOM! they're there. It also gives us what I believe is the one true reason Apple invented the 30 inch Display. ;-)

The 30 Inch Display: Fulfilling the measure of its creation.

ARD displays 50 machines at a time and when you have a capable machine, it uses the "cube rotation" effect to move from one group of 50 to the next. I got a picture of the effect mid rotation below:

Rotates like butter!

So how does it all work? Like this: On each machine we have two volumes: ChangeOS and Mac OS X. The Mac OS X volume is where we install the different versions of the OS. We boot to the ChangeOS volume to free up the Mac OS X volume for modification. When we trigger an automation run we specify the OS version and language. Each machine then reboots to the ChangeOS partition, caches the OS .dmg locally and uses the asr command line tool to restore the image. The tool that does this work is one I wrote (in AppleScript Studio no less!) called Lab Assistant. We have images of the Mac OS from 8.1 all the way up to 10.4.6 in all the languages our products support. It's a lot of data which brings me to the backbone of our automation system:

Our Xserve RAID and XSAN

1 TB

Right now we've just been testing out the XSAN stuff to see how we want to use it. That's why you see all the Xserves. Just one side of the top Xserve RAID is 1 TB of data. For a fun comparison this whole rack which is about 7 ft tall is full of old RAID arrays is also 1 TB of data storage. We call it the Big Mac Daddy.

Big Mac Daddy

Other groups at Microsoft have hardware retention policies that force hardware upgrades every so often, but instead of just "recycling" these server machines, our Lab Manager intercepts them on the way out, and we use them for various things, storage, SQL server etc. We actually have some of the old MSN servers in our Mac Lab!

Hallways

When you have so many machines to maintain, being able to get behind the machines is very important.

Lots o' cables!

Top View

We like to pack in those Mac minis and the cords get pretty dense when we do. The hanging Mac mini box moves if the HVAC is working. If it's not working, we've got to turn off the machines until it's fixed.

Our main automation Xserve has a habit of failing in some serious way once a year, always around Christmas time. :/ For the last 2 years I've been in charge of fixing it and getting it back to operational. Most of our server racks are generic white enclosures, but we do have 1 black Dell rack. As punishment for bad behavior, we put the Xserve in the Dell rack. That'll teach it. ;-) This is what it looks like from the inside of the rack looking out on the world. Poor caged servers...

From the server's perspective

Of course our iWork/iLife balance is just fine as you can see by the following:

Tornado Foosball Table

One of our team members bought this awesome Tornado foosball table which we use along with and XBox and XBox 360 to relax after a hard days work.

Published!

A while back the Seattle PI actually did a front page story on the Mac Business Unit and you can see from the picture in the Lab it was when we had only the G3s.

Gotta have the Mission Statement

As you enter the kitchen we have our MacBU mission statement to remind us what it's all about. :)

Free drinks of course!

Just like everywhere at Microsoft, we get all-you-can-drink beverages.

Fun Times!

Part of our team mantra is that we work hard, and play hard. So we do lots of fun morale events. We just take time off work and do stuff. We're good friends and enjoy "just hanging out" together.

The Mac Library

This is just a pretty picture to represent what is really a much bigger collection of 3rd Party software we use to test with Office. Most if it is stored on file servers, but this gives you an idea. (There's some old WWDC DVDs if you can find them!)

Printer Lab

More Printers

All connected for printer testing

They're beautiful

A big part of Office functionality is printing, and we do loads of print testing. We work really closely with the printer vendors and make sure the printed page looks great. WYSIWYG is fundamental to Mac ethos. All these printers are connected via USB hubs and Ethernet to a Mac OS X Server 10.4 which is the printer server.

I hope that gives you a better idea about what the Mac Lab is like and what it's like to work in the Macintosh Business Unit at Microsoft.