How Microsoft does
“Software
Engineering”
Myung Ho Kim
National Technology Officer (NTO)
Microsoft Korea
Overview
ProductGroup Organization
Engineering Software Products
Hard Decisions and Wisdom
Product Group Organization
Group
Division
Feature-set
Product Unit User Ed &
Product Unit
Marketing Localization
Program
Management Test Dev
Architecture
Implementation
Build BVT QFE
Testing
Quality assurance
A ‘Typical’ Product Group
25% Developers
45% Testers
10% Program Management
10% User Education / Localization
7% Marketing
3% Overhead
The Developer Division
(DevDev)
Has about 2,200 people
Visual Studio / .NET Framework: 30+ Product Units
.NET CF .NET Client .NET SDK
CLR ClickOnce DDUE
DDCPX MSDN NCL
PSDK Setup DataWorks
Team System TS: Architect TS: Dev/Test
(Overall)
TS: Team Foundation Visual Studio for Office VB
VC# VC++ VJ#
VS Core VS Data VSDE
WebData XML WebPT WMI
XES xM Localization
DirectoryServices
VS 2005 & .NET Framework 2.0
Over 8 million lines of code
Over 15,000 files
Over 150 binaries
Typical Project Cycle
Vision Feature RTM
statement complete
Releas
Plannin M1 M2 e
g Testin
g
Technology Beta 1 Beta 2 RC1
Preview
Planning Phase
Marketing gathered requirements.
Senior leadership provided an initial vision.
A cross discipline team created a vision
statement and elaborated on product
themes.
The same team created an overall
schedule.
Product Units began a detailed planning of
what features would get into the release.
Scheduling *
Milestone = unit of product cycle progress
– Enable accurate assessment of progress and distance left
Milestone features are scheduled in priority order
– Enable flexible scheduling to respond to feedback later
Milestone is done when quality exit criteria is met
– Forcing function to ensure team doesn’t go fast and loose
– Ensure very stable, usable, and complete product at exits
Key Feature Milestone Events
Milestone Event Definition
Spec Complete Date design specs for milestone features written &
reviewed
Feature Coding Typically 8-9 weeks in length for feature milestones
Code Complete (CC) Date all features scheduled for the milestone are finished
Test Plan Complete Test plans for all milestone features written and reviewed
Test Case Code Date all test cases in the Test Plan are completed
Complete
ZBB Test Pass (ZBB TP) All existing functional tests are run on current builds
Zero Bug Bounce (ZBB) # of milestone bugs older than 48 hours = 0
Zero Resolved Bugs # of resolved milestone bugs to be verified before closing
(ZRB) =0
Test Sign-Off Final verification and media sign-off on the milestone
build
Design Specs
The feature-set is driven by Program
Managers
– Own the customer experience and making the
right thing happen
– Drive the product unit team during the design
process
– Own writing a design specification for each
feature
Coding
Developers use a variety of tools for coding.
– Visual Studio, Emacs (huh!), vi (huh!!), Source Insight…
– No tool-specific files or code-injection allowed to be checked-in
Maintain single source tree for entire product line.
– Can type “build -c” from root to build from scratch
– Can type “build -c” in any sub-directory of source tree
Each team maintains a private branch off main tree.
– Use internal source control system optimized for
branching/merging
– Minimizes check-ins into main tree and avoids breaks
Coding
C++ & C# are the 2 common languages used.
The goal is efficient, clean, and maintainable
code.
– Code we ship lives at least 10 years guaranteed
support
Documentation and resource strings are
stored externally from source code.
– Avoid documentation writers colliding with
developers
– Localization work handled centrally in Ireland &
Japan
– .NET Fx is localized into 34 languages, VS 2005
into 8
Coding
All code check-ins must be reviewed by
another developer on the team prior to
submitting any changes to the source tree.
Dev is responsible for “check-in suites” to
exercise implemented features.
All product check-in suites must run and
pass 100% before any check-in can be
submitted into source tree (2-3 hour
process)
“Check-in mail” is sent to the team
summarizing changes made, bugs fixed, and
files modified.
Producing Daily Builds *
A new “build” is built and released every day.
– Forces engineering discipline, critical in the end-game
– Build number indicates build date (40730: Jul. 30th )
Central build lab produces daily builds for entire
division
– Staffed 24 hours a day throughout the week
Build process:
– Build team syncs source code tree (~mid-night)
– Kick off end-to-end build (~4:00am)
– Build complete (~1:00pm)
– Perform BVT run to verify build works (~4:00pm)
– Pick up hot-fixes & re-build for BVT failures
– Repeat hot-fix/BVT cycle until the build passes clean
Testing
Test team is staffed by Devs
– Designs test plans, writes tests, and builds infrastructure
Target goal with tests is to reach >70% product code
coverage by end of the product cycle
– We measure this statistic throughout the product
– In Visual Studio 2005 - 62% (native) and 70% (managed)
The Visual Studio 2005 test status:
– 10,339,207 functional test cases
– ~9,000 servers in test lab to run them in automated way
– A full test pass takes 21 days
Test management system manages test plans/runs
“Mad-dog” Test Automation
System
Bug Tracking
Bugs & work-items are tracked within an
internal bug-system “Product Studio”.
– Enables rich reporting and change history
tracking
Bugs triaged by feature leads and assigned
priority
– Bugs fixed in priority order of range from 0 to 4
Daily status mails sent to entire division
tracking bug status
– After the final feature-milestone is completed,
life of the product team revolves around driving
it to zero.
Ship Mode: Coasting In
Large projects “coast in” r.t. ending suddenly.
– You can’t dock a large ship abruptly.
Key set of steps along the “coasting path”
– Lock feature-set and stop adding/changing design
– Run a full test pass to find all bugs of the locked
design.
– Push for a zero bug bounce (ZBB) on the locked
design.
– Absorb bug-bounce over few weeks from ZBB.
– Triage all bugs out of the system that are not “must
fix”.
– Enter End-Game mode & start minimizing code
churn.
End-Game Mode
“War Team” takes over to the End-Game mode.
– Made up of most senior team members, all with ship
experience.
– Guides the ship in into port.
– Meets at least once a day, ≥ twice a day at the very end.
Over the span of the final weeks before a release
– War team steadily raise the triage bar, and towards the
end only allow “showstopper” bugs to be fixed.
– If product is ready to ship, the number of showstopper
bugs will slowly decline as check-in rates drop.
End-Game Mode
Shipping is more of an art r.t. a science
– Senior leaders need to feel that the product is right,
have confidence in test coverage, customer
scenarios…
Once the rate of incoming bugs gets to zero, put
the build in escrow for a few days, and wait &
see.
– If it holds, ship it.
– If not, repeat until we are finally there.
Milestone ends
Rinse and repeat (Iterative process!)
– Visual Studio 2005 went through M0 (planning), M1,
M2, Beta 1, M3.a, M3.b, M3.c, Beta 2, RTM
From Bugs to Builds
Component Dev coding & Buddy Code
triage Unit testing testing check-in
Daily
Bug found
or escalated War Team build
Build CDs BVT
available RTW Pkg Dogfood
Customer Connections
We realized that our customers need to be
part of our development.
– We had to break the pattern that let us think
that “we know what’s best”.
Four areas of focus:
– Product Feedback Center (Ladybug)
– Self-sustaining Communities (Forums)
– Community Technology Previews (CTP)
– Technical Assistance Program (TAP)
Refer to site
http://blogs.msdn.com/ddcpxblg
Hard Decisions
The Golden Rule
– Features + Quality = Time
You can only control two from these three!
Development focus had (always) been Q &
F.
– Don’t worry too much about the schedule
Front Loading Quality
– Maintain a very high quality standard
throughout the project to mitigate F/Q/T
tradeoffs
– No feature work items get checked-in until
they are completely passing tests; there is
always next release
More Wisdom of the Ages
Products ship with bugs...
– The last bug is found when the last customer stops using
your product
People will always screw up a good project
– Don’t add them unless you really have to
– “Mythical man month”
The world is not so static from specs thru RTM
– Resources, priorities, markets, features… will change
– Focus on what ultimately matters
• Quality
• Compelling to customers
• Solving business problems
A Decade of Advances in Software
Construction
and Microsoft’s Scorecard
1. Design has been raised a level
2. Daily build and Smoke test
3. Standard libraries
4. Visual programming innovation
5. Nice community of programmers
6. The Web, for research
7. Widespread use of incremental development
8. Test-first development
9. Refactoring as a discipline
10. Faster computers
-- Source: Construx, Code Complete 2: Realities of Modern Software Construction, 2004
Software is built by People
Understand what’s really bad for
morale
– Long hours and high stress
– Unrealistic schedules
– Feeling of low productivity
– Working on poor-quality software
Software is built by People
Watch out workplace demons!
– Dishonesty
– Sloppiness
– Backbiting
– Intolerance
– Jealousy
– Sexism
– Carelessness
– Inability to speak to power
– Lack of respect for the other
– Blaming
Please come out from the color of the
night…
Closing Remark
Factors that impact any successful
project
– Technology
– Process
–People
