David Campbell is the
General Manager of Strategy, Infrastructure and
As I was preparing the
questions for the interview, I realized that there were too
To give you a sense of
how much David is appreciated by people who know him,
Q.
I’d like to start with a
question a bit off topic.
It has been almost four
months since Jim Gray was reported missing at sea when
Since you have been
working at Microsoft since 1994 and deeply involved in
A.
Wow, this is a tough
one. Jim was a giant in the field of databases – actually he
These two
characteristics – the ability to see things where others couldn’t, and
Q.
Can you tell us about
your areas of responsibility beyond what everyone can
A.
The last couple of
years have been very interesting. SQL Server has grown
Q.
The most major leap that
SQL Server made over the years was probably the
A.
The architectural
transition for SQL Server 7.0 was a great experience and
By the time we were
redoing the engine for SQL Server 7.0 most database
Now to the Hubble’s
Law thing. Hubble was the astronomer who first presented
The other anecdote is
that there was a time when SQL Server 7.0 was introduced
Q.
Even though each version
of SQL Server has improvements and enhancements in
A.
The areas you list
above are what we typically call “Enterprise Abilities” or simply
SQL Server 2005 is a
major release and is really the first example of the
For Katmai we have 4
major themes:
·
“Enterprise Abilities” – which I mentioned before.
· “Beyond
Relational”, which is about managing “all data, birth to
· “Dynamic
Development”, which is really about dramatically reducing the
· “Pervasive
Insight”, this theme is about bringing Business Intelligence to
Q.
I’m sure that the
Microsoft research group these days are laying the foundations
Can you take us 10 years
forward in time; how do you envision SQL Server in the
A.
I’ll frame my response
by saying that the line between “platform” and
1. The
system will be much more adaptive and scalable. It will be able to
2. The
focus will move to the data itself rather than on the machinery used
3. Most
of the data services provided by SQL Server will be driven from a
4. Finally,
fewer and fewer people will miss, (or remember), the “open
Q.
In the days of SQL Server
6.5 a single human being could aim at knowing SQL
A.
I think the most
diplomatic way to answer this is to look at the predictions above
Q.
Besides technical skills,
what are you looking for at the personal level when
A.
I look for passion,
perseverance, and the ability to work well with others. There’s
Q.
What is the process of
evaluating and adding new features to SQL Server? Has
A.
This has changed
tremendously. When we rebuilt the database engine for SQL
Q.
A common question that I
get from students and customers is “Should all tables
A.
Hey, is this a quiz?
My rules of thumb are:
1. If
you have a table with a single index you should generally make it a
2. Clustered
indexes are typically the way to go but watch out for the key
3. Watch
out for insert rates on clustered indexes with monotonically
4. For
logging tables with no indexes and high insert rates go with a heap
Q.
How many people today
are part of the various SQL Server teams? Can you
A.
The team is about 15
times larger than when I started. We’ve had a number of
Q.
From yours and
Microsoft’s experience, how is SQL Server 2005 accepted, and
A.
SQL Server 2005
adoption has been very good. We met with an analyst who
One of the challenges
we had during SQL Server 2005 development was simply
Q.
Once SQL Server was all
about SQL; today, it’s about many technologies—CLR,
A.
It’s a fair question
and I’d expect nothing less from a T-SQL guru like you! The
We spend a fair bit of
time thinking about what not to put in as well. Very few
Q.
Can you tell us a little
bit about yourself outside of work? What do you like to do
A.
I have two teen age
boys and the youngest one is a junior in high school so my
Most of the
non-technical books I read are about history, psychology or are
Q.
Back in the days when you
were deeply involved in the shaping of the storage
A.
Thanks for the kind
words. I really miss writing about stuff but it takes time for
I have a big queue of
things to write about – hopefully I can jot some of it down
Architecture of Microsoft SQL Server. He joined Microsoft in 1994 and played
key roles in shaping the product that grew and evolved to become the SQL
Server that we know today—the product that many of us dedicate our careers
and passion to. You can find David’s BIO here:
http://www.insidetsql.com#David_Campbell.
many questions that I wanted to ask, so I had to restrain myself.
a colleague of mine who knows him for years told me “David is one of the
most
genuine people you will ever meet.”
he went sailing to spread his mother's ashes. Jim Gray is a well known and
admired figure in the SQL Server community and everyone who knows him feels
devastated by his disappearance. I understand his family has not given up
hope
yet, and we all join them in hope to find any sign of him.
shaping the SQL Server database technology you most probably had many
interactions with Jim. Can you tell us about your interactions with him;
what he
means to you, to Microsoft, to science? Who is Jim Gray the human being from
your perspective, and what was his contribution to SQL Server, databases,
and
science in general as a scientist/researcher?
was a giant in science. I guess what impressed me most about Jim was that he
made
everyone around him feel like a giant as well. He was a great collaborator;
a
great “connector” who brought people together from different companies, even
different fields, to pool their knowledge and make breakthrough advances.
Jim
also had an amazing ability to look at things differently than those around
him.
He could simply see many things that others couldn’t, and he could take
ideas
from one domain and immediately see their relevance in some completely
different domain.
the ability to connect people from across an amazing network, made Jim an
incredibly influential and loved person in many communities. We all miss
him.
read in your BIO, and in a less formal manner - what does your day look
like?
tremendously and the product is now at the point where we can really start
to
innovate. It took 10 years and 2-3 releases to go from a relational database
engine and client libraries to what we refer to as a “Complete Data
Platform”
with Analytics, Data Mining, Integration, Reporting and more. We are no
longer
“chasing taillights” and are now in a position to do some amazing things. I
now
lead a group that we call “SIA – Strategy, Infrastructure & Architecture”
and
spend much of my time working with other teams around the company trying to
chart how we can help advance the data platform vision to solve new and
interesting customer problems. I believe the next major advance in
Information
Technology will come from addressing the gap between people and information.
Microsoft is the best place to deliver on this vision and I often tell
people I have
one of the top 10 jobs in the company.
redesign of both the Storage Engine and the Relational Engine in SQL Server
7.0.
Even today, with SQL Server 2005 the core engines are still very much based
on
version 7.0. This was also the leap from a small database to an enterprise
level
one. Can you tell us what, in your eyes, were the
most important architectural
changes of the engines that enabled SQL Server to become an enterprise level
database? Also, are there any major architectural changes planned in the
engines in future versions that you can share with us?
achievement for all of us who worked on it. If I were to pick two
architectural
tenets that paid off and are still relevant today I’d have to say automation
and
paying attention to Hubble’s law.
servers had 100’s of control knobs and very few people truly understood what
they all did. The few that did know couldn’t turn them fast enough to keep
up
with the dynamic workloads that were emerging in the mid 1990’s.
Furthermore,
we saw many, many systems that were performing poorly because they were
mis-configured. We rethought our philosophy
around the knobs so that instead
of controlling the things we simply couldn’t figure out how to do
dynamically in
the code, (remember “hash buckets”, or “open objects”?), we added knobs to
capture the administrator’s intent. This was a great achievement but we were
ahead of the market when we released SQL Server 7.0 and this was
frustrating.
DBAs were afraid we were going to put them out of work since they were paid
to
change the knobs, typically when their pagers went off. Furthermore, our
competitors were saying, “SQL Server; how can that be a real database
product?
– it has 20 knobs and ours has 500!” Of course, people have since realized
the
value of this work and it is becoming commonplace in the industry.
evidence that the universe is expanding. I believe Pat Helland should get credit
for associating “Hubble’s Law” with the increasing latency or “distance”
between
CPUs, main memory, and disks. In recognition of this effect we completely
redesigned the I/O subsystem and the query processor to try to do deep
predictive I/O forecasting over the disks and try and turn as much random
I/O
into skip-sequential I/O as we could. This is why we introduced things such
as the
Index Allocation Map (IAM) to control and record allocations in a dense data
structure. These changes led to some amazing speedups and a couple of
memorable anecdotes. In the “wide update plans” that we introduced in SQL
Server 7.0 the query processor can do large updates by computing a delta
stream,
sorting it, and then applying the changes in key order to an index thus only
touching each page once and in optimal disk order, (assuming everything was
defragmented). This can dramatically reduce the random I/O for a large
update
and we had someone file a bug when a typically large update query they used
to
run that took over an hour on SQL Server 6.5 ran in less than a minute on
SQL
Server 7.0. At first they thought they were sure they had found a defect in
the
product, but then they checked the data and it appeared correct. Ultimately
they
filed a bug anyway to ask us to explain what was going on.
where it was about the best disk driver stress test on the planet. I
answered
many newsgroup posts that went like this – “I installed SQL Server 7.0 and
now
my machine blue screens”. I’d respond, “I’ll bet you dollars to donuts if
you
upgrade your I/O subsystem firmware and drivers the problem will go away”.
Sure enough it would. There was one hardware manufacturer that admitted to
me that they had seen I/O blue screen in their labs with their tests but
hadn’t
bothered to fix it since there were no real world applications that
encountered it -
(before SQL Server 7.0 shipped).
several areas, I guess that each version has some main focus area
(performance,
stability, functionality, high availability, security, etc.). What was the
focus area
for each of the versions of SQL Server as of 7.0, and what is the focus area
in
Katmai?
“the –ilities”. We need to move these forward
with every release as the demands,
expectations, and scale of database applications continue to grow. Real
quickly
I’d say SQL Server 2000 was about finishing SQL Server 7.0. We initially
thought
we’d need a very quick “point” release to respond to market feedback on 7.0
but
since SQL Server 7.0 was such a dramatic improvement over 6.5 we took a look
after we finished our first round of SQL 2000 features and since 7.0 was
doing so
well we went back in and added some more to SQL Server 2000 before shipping
it.
“Complete Data Platform” I mentioned earlier. Having the Business
Intelligence
services “in the box” has been tremendous. The fact that we have analysis,
integration, reporting, data mining, service broker, and native XML support
in
the product is allowing people to build some incredible end to end
applications.
archival.”
“time to solution” to address information requirements – whether it’s a
new application, analysis over existing data, creating reports to drive the
business, etc.
the masses. Rather than have BI be just about informing the 10’s of
strategic decisions a company makes on an annual basis, we want to use
it to empower the people who are really running the business day to day.
for the future database and that the SQL Server team interacts with the
research
group.
year 2017?
“application” tends to move upwards over time. 15 years ago networking
wasn’t
part of the PC platform and you had to buy hardware and a TCP stack to put
your
computer on the network. 7-10 years ago “Data Mining” wasn’t part of the
database platform, etc. Here are some predictions:
meet increasing processing and capacity demands without a lot of
human intervention. We’ll be much closer to being able to wheel in
compute and storage resources and have the system integrate them into
the mix automatically.
to manipulate it. We’ll be less concerned with the plumbing and more
concerned with data quality, data protection, and information
production.
common data model. Whether you’re creating a report, building an
information cube, or integrating data from another system, you will be
able to start from a common model of the key data entities such as
“customer”, “order”, or “prospect”.
databases” sp_configure option…
Server really well. These days the product is so huge that you can’t master
the
product, rather a certain area of the product. For those who seek to invest
their
time and effort building a career around SQL Server, what would you
recommend focusing on? Which SQL Server areas and technologies do you find
that have great potential?
and draw your own conclusions about how to build a career around SQL Server.
If you believe my predictions you can bet one way, if you don’t, bet the
other.
hiring new people for the SQL Server team?
a “make stuff happen” gene that some people possess that I also try to spot.
I’ve
seen all too many technically brilliant people fail due to a lack of these
characteristics.
anything changed in this process over the years?
Server 7.0, or when the Analysis Services team built AS we knew what we were
going to build. In many cases we had people who had built these things
before so
the playbook was pretty much written. Today we have all the key features and
customers no longer ask us if we have “row level locking”, or “online index build”.
Instead, they are asking us to help solve their business problems and to
help them
do it quickly and effectively. We’re now much more mindful of customer needs
and building end to end scenarios to address those needs. Sometimes it will
take
several releases for the entire picture to become clear. For example, in SQL
Server 2005 we built a great security infrastructure – these are building
blocks. In
Katmai we’re using these building blocks to address customer scenarios like
information leak protection and transparent encryption.
have a clustered index or are there cases where it makes sense to organize
the
table as a heap?” What would you answer to such a question?
clustered index.
size.
increasing keys. (You can distribute inserts sometimes by doing some key
munging tricks…)
as we can distribute the inserts across a number of pages.
share the challenges the group went through and how you overcame those
challenges when changing from a small to such a big group?
challenges in growing so fast and perhaps it’s no surprise in that one of
the
biggest challenges is staying nimble and agile while we grow. We’ve changed
our
development process a lot in Katmai to address this and we’re still having
some
growing pains but one of the things I really like about our team is that
we’re
honest about our challenges and continuously strive to make things better. I
think customers will definitely see the results of our process improvements
in the
polish and scenario completeness in our upcoming release but it will
probably
take another release cycle to get really good at the process and tune it so
day to
day life is efficient for everyone on the product development team.
what is the adoption rate? What major issues did you face, and were these
addressed in Service Packs 1 and 2?
tracks the adoption rate of enterprise database releases and his data showed
incredible uptake thus far.
that there was so much going on in the release and we were changing a lot of
infrastructure. Perhaps the most tension came in the management tools where
the team not only had to move our framework over to the managed shell but
also had to keep up with the features pouring in from the other component
teams. There were strong opinions on SSMS from customers who had to spend
their work day in it but the team has done a great job in addressing
customer
feedback on SSMS in SP1 and SP2.
XML, and so on. Not underestimating the value of the added technologies and
admitting that some important T-SQL features were added to SQL Server 2005
(CTEs, Ranking Calculations, Exception Handling), I do feel that T-SQL
doesn’t get
enough focus and attention. For example, in terms of ANSI SQL grammatical
support SQL Server seems behind. Other database platforms already
implemented the profound OVER clause fully, have support for regular
expressions in SQL, intervals, full support for vector expressions/row value
constructors, more complete support for recursive CTEs (searching and
cycles),
and so on. For proper disclosure I should say that since my focus is T-SQL I
may
be biased and always want more... Am I completely off base here? And if not,
is
it that Microsoft simply believes that the surrounding technologies are more
important?
real answer is that it’s a balancing act. Every component team wishes they
could
do more in each release and every MVP and expert on a particular aspect of
the
product feels like we could do so much more for their area. In reality we’re
trying
to focus on helping our partners and customers solve their business
problems. If
we really need great regular expression support in the engine to do this
we’ll put
it in. Honestly, it’s a return on investment thing and we have to look at
the big
picture across the entire product.
people would consider a Swiss army knife as elegant and once you put in a
feature, even if it’s only used by 2% of the customer base, it’s very hard
to pull it
out again.
when not working; what kind of non-technical books do you like to read?
wife, Marcia, and I are spending time lately visiting colleges with him. I
enjoy
travelling and photography and I credit Marcia with teaching me how important
travel experiences are. We’re trying to visit all the National Parks in the
US –
(currently at 22 out of 58). These trips scratch both my travel and
photography
itches.
biographies of one form or another.
engine of SQL Server 7.0 you used to participate in some private SQL
newsgroups. Your explanations were so detailed and clear and were considered
pure gold by many of us. For example, I remember how you solved a major
mystery that involved querying a small heap that reported a huge number of
logical reads, after the table underwent an update statement expanding
varchar
strings. You explained that the expansion of the
varchar strings that didn’t have
room to expand in their hosting pages caused a large number of forwarding
pointers, and that SQL Server had to jump back and forth between the page
holding the pointer and the page holding the pointed record. Suddenly it all
seemed to make sense. For us teachers and students, someone with both a lot
of knowledge and great explanatory skills is a rare sight, and the SQL
Server
community can benefit from having this knowledge conveyed through books.
Have you considered/are considering writing a book and passing on what you
know through such means?
me to do it well and I have so many other things going on now that I find
myself
frustrated if I try to respond quickly. Just last week I responded quickly
to an
e-mail from a customer I met at an event and instead of giving him the full
“it
depends” explanation with the rationale I gave him the quick conclusion
since I
was falling behind in e-mail. Unfortunately, this confused him and others on
the
thread and it took 2 other e-mail exchanges to sort it out.
before I forget it all!
Interview with David Campbell
2 comments
Hide comments
