|
|
|
Microsoft Beta Testing
 Microsoft Beta Testing: Beta testing has been the cornerstone of the software development process for Microsoft and most other commercial software makers for as long as they've been writing software. But if certain powers-that-be in Redmond have their way, betas may soon be a thing of the past for Microsoft, its partners and its customers.
What's behind the shift?
Beta builds are predictable milestones by which external and internal testers have measured their progress toward the delivery of final code. Betas are typically few and far between, and feedback on them tends to be late and sometimes lame. But they are nonetheless a known quantity, and one that software developers, OEM partners and customers of all sizes rely on in gauging when and how to support a new product.
Open source turned the traditional software
development paradigm on its head. In the open source world, testers
receive frequent builds of products under development. Their
recommendations and suggestions typically find their way more quickly
into developing products. And the developer community is considered as
important to writing quality code as are the "experts" shepherding the
process.
Until fairly recently, Microsoft championed the way it "made the
sausage" as the best way. But in the last year or so, there's been a
noticeable change in Redmond's rhetoric. Top Microsoft brass have been
wondering aloud whether big-bang betas are the best way to develop and
test software. Might there be a better alternative?
Microsoft's Engineering Excellence unit, headed by 20-year company
veteran Jon DeVaan, is overseeing a project to reengineer the way
software is developed inside Microsoft. DeVaan, a member of Microsoft's
central Business Leadership Team, manages the engineering standards used
to create Microsoft's software products.
With feedback from the Windows Core Operating Systems
Division (COSD), Office, development tool and applications units, DeVaan
is spearheading a fundamental change to its engineering process. As part
of this initiative, Microsoft's major product units will do more
"virtual teaming" across division lines, bringing together developers
from multiple units to build a new product or subsystem, and then
disassembling them again once the project is completed. At the same
time, these teams will rely on CTPs (or whatever Microsoft ultimately
ends up calling the more frequent test builds) as opposed to betas, to
speed up the product development and testing cycles.
In January, Jim Allchin, co-president of Microsoft's Platform Products &
Services Division, hinted at this change. Allchin acknowledged that the
Vista team was relying on more frequent CTP test builds, as opposed to
traditional betas, in its final development stages for Windows Vista.
The SQL Server team also favored CTPs over traditional beta releases in
the final throes of SQL Server 2005 testing and development, said Paul
Flessner, Microsoft senior vice president of Server Applications. In
fact, the SQL team decided to forego Beta 3 of SQL Server 2005 because
developers felt the CTP feedback they were receiving was solid and rapid
enough to push product out the door, Flessner said.
Going forward, if Flessner prevails with what he describes as the "SQL
Server Reengineering Initiative," it's going to be a beta-free world for
SQL Server developers and testers.
"If I had my way, there'd never be another (SQL
Server) beta," Flessner told Microsoft Watch. "The boss doesn't always
get his way, but we are definitely leaning toward this."
This change of heart isn't coming out of the blue. The open source
development process has influenced Microsoft's thinking, Flessner and
other Microsoft officials have admitted. But there are other factors at
work, as well.
First and foremost, Microsoft divisions that have piloted CTP programs ?
primarily the Windows, SQL Server and Visual Studio units -- and its
testers have found CTPs to be a more efficient way to exchange with
testers code and feedback on it.
In the last 18 months that SQL Server 2005 was in development,
Flessner's team "really got in the CTP groove," he said.
"CTPs helped me keep the engineering team in a branch where we could
ship," said Flessner. "It kept the team from checking things in until
they were of high quality. We had a very sophisticated branching system
and very high-quality reverse integrations that got integrated back into
our main builds. That was a super-powerful thing."
With "Katmai," the next major version of SQL Server, Flessner is
planning to move to a pure CTP/no beta model, he said.
Granted, what the SQL Server team labeled a "CTP" was different from
what the Visual Studio or Windows team labeled a CTP. The Visual Studio
team, which established the de facto definition of CTP as a "snapshot of
a product at a given point in time," released CTP test builds that were
not of beta quality, officials acknowledged. The Windows team
complicated matters, by releasing December, Febraury (and soon, April)
CTPs of Vista that officials also described as Beta 2 builds. The SQL
team, for its part, did full quality checks on each and every CTP it
issued last year.
Because of the differences in terminology around CTPs, some kind of new
nomenclature is likely for these test builds, Flessner said. Regardless
of the label, however, the impact will be similar to that of CTPs as
they exist today, he said.
"When a feature is ready, it will go into the product," Flessner
explained. "I'd like to be in a situation where we can ship (a new
release or update) in 120 days, due to market requirements, the team
being tired or whatever."
Flessner said he expects the new engineering process to have an
immediate and beneficial effect on product ship cycles for everything
from the next version of SQL Server, to WinFS, the Windows File System
code.
"I'll be able to go to a 24- to 36-month (new release) cycle," he said.
"I couldn't do it without these engineering changes."
Based on that timetable, SQL Server Katmai would be a 2007 or 2008
deliverable. And SQL Server 2005 Service Pack 1 ? which will include the
data-mirroring technology that Microsoft cut at the last minute from the
SQL Server 2005 product due to quality concerns ? will also come to
market more quickly. Microsoft released this week its first CTP for
Service Pack 1. Flessner is estimating the final service pack code
should be ready in April ? many months quicker than the typical service
pack is developed and tested.
Flessner said the team is aware there are refinements to make to the
process. Customers have reported trouble attempting to mix and match
different test builds of Microsoft products.
The ability to install the latest CTP atop the most recent one is a
priority, as well. "Customers need to be able to upgrade. When you can't
upgrade from one CTP to the next, that is a definite problem," Flessner
admitted.
At the same time, Microsoft is thinking beyond how and when to deliver
the bits and is also reworking its processes, Flessner emphasized.
"There's the whole question of what constitutes a milestone any more,"
Flessner said. "It's very much like how Boeing is building planes using
sub-assembly teams that are multi-disciplinary."
But there's one key difference, Flessner admitted. "We get to fly each
thing with each CTP. That's is our advantage."
|
|
|
