19 April 2006

A Tour of Microsoft's Mac Lab

Today I'll stop my 3rd person perspective. I'm going to write a little bit more about what I do to help MacBU ship great software and provide some color around what's it's like to work on Mac software at Microsoft. Often when we have press events or special visits from our MVPs I'll give them a tour of the Mac Lab and explain what we do. They've always found it very interesting and so I thought I'd share a virtual tour of our Mac Lab. Let's get started:

Lab Door

We start with the door. The Mac Lab is about 2000 square feet of solid computers. The calendars you see down the side of the wall we use to mark team birthdays as well as special events. The CD on the door is an old Apple Software Restore CD from the last beige Mac Apple produced, the Power Macintosh G3.

The "Sandbox" and Plasma display

The first area in the Mac Lab is what we call the Sandbox. This is where we keep all significant hardware configurations Apple has released that run our products. We'll use the Plasma display to, watch DVDs and play games, uh er, I mean, do important training presentations. ;-) It's actually very useful because everyone can be in front of a computer and still see the main screen and follow along. Often other groups at Microsoft (the games group, hardware drivers group and even the Windows media group) will come and schedule time in the Mac Lab to test their software on the different hardware configurations.

Old iMacs and iBooks in the distance

More Macs

We have lots of Apple hardware. You can see here the old colorful iMacs along with some of the old iBooks. You can also see two of our Lab Technicians working on the backup systems, but more about that later. Up until a few months ago we had every significant hardware revision Apple has ever released since the dawn of time. We even had a section of the Lab we affectionately called the "Mac Nursery" where we kept all the older Macs going. We even had an old SE/30 and IIci and super expensive Mac II all connected via PhoneNet running Spectre, just for fun. It's always super fun to boot Word 1.0 or Excel 1.0 on these old machines and see how much things have changed. Due to lack of space in the Lab we had to put all of these older machines into storage and recycled the following:

  • Macintosh (original)
  • Macintosh SE
  • Macintosh SE/30
  • Macintosh Classic
  • Macintosh Centris 610 <-- I had this one in my room during high school
  • Macintosh IIci
  • Macintosh IIsi
  • Power Macintosh 7100/66
  • Power Macintosh 7100/80
  • Power Macintosh 7500/100
  • Quadra 650
  • Power Macintosh G3
  • Duo Dock with Powerbook Duo 2300c
  • Power Computing PowerCenter Pro 210

May they rest in peace.

ADIC Backup Robots

One of the realities of working with computers is that things fail. More often then you might think. We've used different backup robots, but ever since we moved to ADIC we've never had a robot failure. They just make great stuff. We have 3 robots and use Veritas Backup Exec. It works pretty well, as you can see:

Lots of backup tapes

We also do offsite backups just incase "The Big One" hits, but for regular use these tapes work just great, except when they don't. Recently we had a failure and lost 400 GB of data! We restored it from the tapes and then discovered that the Mac version of the Backup Exec agent had a corruption bug causing the restores to be compromised! Veritas folk were super responsive and they should have a new Mac agent out soon. Backup software and file systems are in the class of software that simply must work, all the time. Alas, this is not always the case.

I'm going to skip the "Build Lab" section of our Lab since it's very much in transition. Maybe I'll post about that later. For now, on to our automation system!

Mac Office is one of those "software in the large" projects. There's really no way a team of our size would be able to adequately test all of Office without the use of automated testing. Every day we get a new build of Office from the build machines, we copy it to our Xserve RAID connected to our dual G5 Xserve for access by our 249 automation machines. We then run thousands and thousands of tests on the new build. Typically we get 4 builds of Office each day: English Ship, English Debug, Japanese Ship and Japanese Debug. We run our entire battery of tests against all the builds and then report any failures to testers via email. The testers investigate the failures, log any bugs and then move on to their other duties as testers. This turns out to be very effective, if used properly, and over time it allows testers to focus on things humans do best, while letting computers verify the repetitious and mundane, but necessary, testing. It all started with our Blue and White G3s years ago. At first when testers would upgrade their test machines, instead of recycling the machines, "The Lab" would get them to add them to our automation machine pool. I think we had about 20 machines to begin with.

Venerable Old Bluies

After some time we started getting Gray G4s...

Speedy G4s

Then we upgraded to some dual proc machines...

Dual G5s

Then Apple give us a special gift. :-) You'd be probably be very surprised at the cost of running all these machines. There's the obvious electricity costs, but also cooling costs and even the physical space costs. Additionally, our system scales, not with CPU horsepower, but with quantity of machines. Most of the tests we run don't run significantly faster on a dual G5 vs. a single G4. So when Apple announced the Mac mini it wasn't minutes before we were considering how to use it for our automation system. The Mac mini has all the perfect qualifications:

  1. Low power
  2. Low heat
  3. Small
  4. Easy to pack together
  5. Inexpensive

So we got a few to test things out...

Mixed Automation Rack

And then we made the big purchase:

Rack A, B and C = 150 Mac minis!

These work extraordinarily well. You might wonder how we control all these Macs. We use two methods: KVM switch box and Apple Remote Desktop. Thanks to our Lab Manager's great relationship with the IOGear folk we have a very reliable solution these days. It seemed like it took for ever to find a USB KVM switch box that didn't leave the machines "headless" after random reboots. The 8 port USB KVM from IOGear has been rock solid. So what does it look like to sit in front of 64 Mac minis? Like this:

64 Mac minis: 1 keyboard, monitor and mouse

This works very well when you must access the machines physically. Even so, just scanning each Mac for 1 second gets very old, very fast and Apple Remote Desktop comes to the rescue! When we need to see all the machines at once we just select them and BOOM! they're there. It also gives us what I believe is the one true reason Apple invented the 30 inch Display. ;-)

The 30 Inch Display: Fulfilling the measure of its creation.

ARD displays 50 machines at a time and when you have a capable machine, it uses the "cube rotation" effect to move from one group of 50 to the next. I got a picture of the effect mid rotation below:

Rotates like butter!

So how does it all work? Like this: On each machine we have two volumes: ChangeOS and Mac OS X. The Mac OS X volume is where we install the different versions of the OS. We boot to the ChangeOS volume to free up the Mac OS X volume for modification. When we trigger an automation run we specify the OS version and language. Each machine then reboots to the ChangeOS partition, caches the OS .dmg locally and uses the asr command line tool to restore the image. The tool that does this work is one I wrote (in AppleScript Studio no less!) called Lab Assistant. We have images of the Mac OS from 8.1 all the way up to 10.4.6 in all the languages our products support. It's a lot of data which brings me to the backbone of our automation system:

Our Xserve RAID and XSAN

1 TB

Right now we've just been testing out the XSAN stuff to see how we want to use it. That's why you see all the Xserves. Just one side of the top Xserve RAID is 1 TB of data. For a fun comparison this whole rack which is about 7 ft tall is full of old RAID arrays is also 1 TB of data storage. We call it the Big Mac Daddy.

Big Mac Daddy

Other groups at Microsoft have hardware retention policies that force hardware upgrades every so often, but instead of just "recycling" these server machines, our Lab Manager intercepts them on the way out, and we use them for various things, storage, SQL server etc. We actually have some of the old MSN servers in our Mac Lab!

Hallways

When you have so many machines to maintain, being able to get behind the machines is very important.

Lots o' cables!

Top View

We like to pack in those Mac minis and the cords get pretty dense when we do. The hanging Mac mini box moves if the HVAC is working. If it's not working, we've got to turn off the machines until it's fixed.

Our main automation Xserve has a habit of failing in some serious way once a year, always around Christmas time. :/ For the last 2 years I've been in charge of fixing it and getting it back to operational. Most of our server racks are generic white enclosures, but we do have 1 black Dell rack. As punishment for bad behavior, we put the Xserve in the Dell rack. That'll teach it. ;-) This is what it looks like from the inside of the rack looking out on the world. Poor caged servers...

From the server's perspective

Of course our iWork/iLife balance is just fine as you can see by the following:

Tornado Foosball Table

One of our team members bought this awesome Tornado foosball table which we use along with and XBox and XBox 360 to relax after a hard days work.

Published!

A while back the Seattle PI actually did a front page story on the Mac Business Unit and you can see from the picture in the Lab it was when we had only the G3s.

Gotta have the Mission Statement

As you enter the kitchen we have our MacBU mission statement to remind us what it's all about. :)

Free drinks of course!

Just like everywhere at Microsoft, we get all-you-can-drink beverages.

Fun Times!

Part of our team mantra is that we work hard, and play hard. So we do lots of fun morale events. We just take time off work and do stuff. We're good friends and enjoy "just hanging out" together.

The Mac Library

This is just a pretty picture to represent what is really a much bigger collection of 3rd Party software we use to test with Office. Most if it is stored on file servers, but this gives you an idea. (There's some old WWDC DVDs if you can find them!)

Printer Lab

More Printers

All connected for printer testing

They're beautiful

A big part of Office functionality is printing, and we do loads of print testing. We work really closely with the printer vendors and make sure the printed page looks great. WYSIWYG is fundamental to Mac ethos. All these printers are connected via USB hubs and Ethernet to a Mac OS X Server 10.4 which is the printer server.

I hope that gives you a better idea about what the Mac Lab is like and what it's like to work in the Macintosh Business Unit at Microsoft.

254 comments:

«Oldest   ‹Older   201 – 254 of 254
Anonymous said...

Great Lab pictures... fantastic insight into Microsoft - which is I am sure not the large evil corporation which everybody thinks it is.. Great Products produced by ordinary people. Looks like a cool place to work.

Nick/

Anonymous said...

cool! thanks :)

Anonymous said...

Right on! I'll trade you foosball lessons for a G5!

; )

Anonymous said...

Hey that was a great tour. Thanks for putting the time into it for us. 'Been an Office user since the dozen 5 1/4" version. Looking forward to your U-Binary version.

Anonymous said...

Wow what an interesting read.

How did you manage to wire all the mini macs to an 8 port KVM box?

Anonymous said...

Wow what an interesting read.

How did you manage to wire all the mini macs to an 8 port KVM box?

Anonymous said...

"How did you manage to wire all the mini macs to an 8 port KVM box?"

If you look carefully at the picture with the KVM in it, you can see there are actually 9 KVMs stacked upon one another. That's 8 KVMs x 8 Mac minis = 64 computers, and one KVM to switch between whichever KVM has the computer you want to look at.

I myself am using a similar system (albeit on a much smaller scale) in my room to hook up my various devices to my single monitor.

Victor Barreiras said...

This is very strange. But how could Microsoft develop their applications without testing them on other platforms? However, why so many Mac Minis, G5s and G4s? It looks more like Microsoft has a little (sweet mac user) devil inside, and the only place to have fun in Greyland-PC is your Mac Lab. I wish I had at least one of each Mac. Can't you give some away?...

Victor Barreiras said...

This is very strange. But how could Microsoft develop their applications without testing them on other platforms? However, why so many Mac Minis, G5s and G4s? It looks more like Microsoft has a little (sweet mac user) devil inside, and the only place to have fun in Greyland-PC is your Mac Lab. I wish I had at least one of each Mac. Can't you give some away?...

Anonymous said...

nice, that looks fantastic. These sorts of things always interest me. I used to run an old 20Mhz powermac as the webserver for my website photogabble.co.uk up until late last year infact. had no problems with it whatsoever, now with the new p4 server that was given to me it crashes about three times a year - not a software issue either, somthing on the motherboard keeps dieing...

Anonymous said...

From now I think different about MS ;-)
Thanks for the tour!

Rob

Anonymous said...

best lange site:P

Anonymous said...

Sweet, thanks for the tour!

http://www.macshadows.com

Anonymous said...

absolut nice photos :)

thanks

Anonymous said...

In the heart of darkness, a kernel of light.

Anonymous said...

Damn you guys! Wish my mac and I were there daily...

Anonymous said...

This is great, know thine "enemy" is of course one of the important tactics! I own 2 Powerbooks, ipod etc AND a Windows PC. Every Windows user ought to get mac stuff too. Cheers, kel at www.apple-music.co.uk.

Anonymous said...

WOW! I wish I had something like that in my bedroom! :P

Anonymous said...

Nice...though I was wondering how you manage the power these. Cause if you just do a sizing for the 150 minis you need something around 50kva if you estimate the power supply is 200watts/mini. If you add switches, servers and so on, how can you manage the all structure? And also how you deal with the cooling issue?

Anonymous said...

coool :) ms isnt that bad hehe

Anonymous said...

Thanks for the inisght into MS Mac Unit. You have my undying thanks for making my life easier. without you, people would bring me word files that I couldn't view, etc. Thank you.

Anonymous said...

Thanks Friend this was a great Tour

Epsi said...

Woah!!!! 0.0 Computer-nerd paradise! ^_^

Anonymous said...

Just plain fun to be invited. Keep up the good work.

:-j

Anonymous said...

Just plain fun to be invited. Keep up the good work.

:-j

Anonymous said...

David - Did the Mac BU get any of the G5s that the gaming division is selling off now that they are done with the XBox 360 initial developemnt? If so, how are you using them, and if not, why?

Thanks - great look behind the scenes.

Rob

Anonymous said...

Looking foreward for the next article!

Anonymous said...

You made my day. You inspire me to go to university and become a computer developer. :) What education and how did you get involved there?

thomasvendetta said...

When you 'recycle' them, do you guys actually throw them out or send them somewhere..? or do you guys just store them in a storage facility?

Next time your recycling any old macs, could you guys contact me? I would be more than willing to pay for shipping and handling if you guys would recycle one to me :-D

Tom

Michael said...

This is awesome! Do you guys have any Powermac G4 Cubes? If you do, would you be willing to share some info about the ones you have? Cubes are pretty cool!

Ditto to Tom, lemme know if you guys are "recycling" any!

Thanks,
Michael

Anonymous said...

I've never seen so many macs in a room....amazing...

bye
fede

Anonymous said...

that was soooo explictyi

Anonymous said...

Nice facilities. now how about using them to make a worthwhile MSN client for OS X?

Anonymous said...

I looked at the photos (then I didn't read the screes of comments) - thanks for the tour.
I have to say it is sad to see such a boring uninspired work environment.
Also sad to see that one of the richest companies in the world offers "all you can drink" total rubbish for your body.

I came to look hoping to see something radically different that would somehow go to explain such a "successful" company, what I see is just a boring average everyday kinda office like the majority of others.

Anonymous said...

Your expectations were off, Shine. Maybe the MacBU is the coolest part of Microsoft, but Microsoft is no longer a special organization overall and that is going to bleed into the MacBU. The culture of Microsoft is the culture mediocrity, of least common denominator. Look at all their weak products. The fact that the MacBU is the only group with any innovation doesn't mean they have control over their workspace.

Microsoft all think they are special but that's because they all share this self-righteous Kool-Aid amongst themselves and tell each other how much better they are than everyone. I have to always point that out because I am sick of the snotty attitudes from everyone who works there, who are so difficult as a client to us, and who are so condescendingly "nice" outside of work.

As you described, there's nothing special going on in there. No cool interior design, nothing modern, just mediocrity. And a ghetto poster of the mission statement tacked to a wall. Sad.

Funny enough, I love the MacBU and really appreciate all the work they've done for Mac. They have turned around my attitude towards Microsoft as evil. Now I just think the MacBU is great and the rest of the company has potential but is mired in ineptitude, committees, fear, stupidity, arrogance, and a lack of talent, not in engineering (except security), but in product development/marketing, and oh yeah, ape-man Ballmer who keeps making bold proclamations about greatness and great products but delivers nothing.

Anonymous said...

Awesome. Love the free drinks. :D

Anonymous said...

One of the rare interesting stories accompanied with the pictures on Interent. Thank you!
Yes, blog should have meaing!
I would offer you more advanced technology for virtual tours - right up your street, sorry, lab. Look what I have: http://www.virtuar.com/click.htm
- San Francisco, Las Vegas just as examples. Also many other small placed including Stereo pictures. No glasses needed.
Igor Polk

Mitchell Scott said...

Mac Pro picture update maybe ;-) ?

Anonymous said...

Wow, a very nice tour. It sorta puts a tear to my eye knowing you're home to so many macs.

Anonymous said...

the evil empire uses macs... figures...

when is mac os X vista version due out now? lol

Anonymous said...

So which do you prefer... honestly...

Mac or Windows?!

David Weiss said...

(In case it wasn't clear... I just can't imagine myself ever working on Windows and really enjoying it.

Anonymous said...

How many testers does the MacBU have? What's the ratio of testers to developers?
Thanks.

Anonymous said...

the power consumption must be huge, even with the mac minis. makes me glad to see microsoft with all the alternative energy sources like solar panels. such energy efficiency is something i'd expect to see at a place like... well.... apple. haha.

Anonymous said...

So which is highly recommended Mac or Windows?

Anonymous said...

Did I spy a "MS Office 2001 for Dummies" book on the bottom shelf of the MacBU lab bookcase? Tsk, Tsk. Seems there would be the MS support you'd ever need just down the hallway. Anyway, great fun. Thanks for the post.

Unknown said...

Can annyone tell my wy the dont use tape Free backup?

B2D Backup to disk

A lot off tapes pfffffff


Tapes Sucks!!!!!

Anonymous said...

These comments have been invaluable to me as is this whole site. I thank you for your comment.

Unknown said...

Brilliant! Going to work must feel like coming home...*sigh*

Anonymous said...

great post ibook rules

Anonymous said...

Thank you for posting this blog on work at the Mac BU unit for Microsoft. I learned a lot about the working atmosphere of a mac unit in Microsoft and I appreciate that you did this.

Tracey C

Anonymous said...

I need to be in that 'jungle ;)

Unknown said...

Wow..
I am jealous of u now ;)
I have been working on mac since last 3 years. But i never imagined a mac lab of this size. ( 240 machines to test automation for Office!!!! )
I will see if i ever get to work for a great product like MS-Office + my Fav Mac OS on lovely apple hardware.

Anonymous said...

Woah! I want your job!
Thats alot of computers...and printers, they really are beautiful...
Looks great!
Thanks for posting this! I really enjoyed it!

Cheers,
Sean

«Oldest ‹Older   201 – 254 of 254   Newer› Newest»