General Network Error - MS Stumped

I've been working with Microsoft support for over 3 weeks now on an
intermittent General Network Error we're seeing in our production
environment between our ASP.NET application and SQL Server 2000. They are
continuing to work on the issue, but it seems as if our progress is grinding
to a halt. I asked this question on the newsgroups before going to MS but
figured I'd give it another shot now that I have more information.
The problem occurs intermittently (maybe 10 to 15 times per day). It seems
to be related to queries that return "large" amounts of data from SQL. The
problem started after we made an equipment change. We went from a single
server hosting both our ASP.NET application and SQL Server 2000, to one
server hosting IIS and another machine running SQL. An example of the
exception thrown appears at the bottom of this post.
MS had us perform 3 data captures initially: MPSRPT_MDAC on the IIS machine,
MPSRPT_MDAC on the SQL machine, and SQLDIAG on the database server. The MS
rep found nothing too out of the ordinary with our configuration, so he
asked me to do a network capture on both machines while the error occurred.
I was able to get a good capture in a small window and sent it off to MS.
The verdict is that IIS is basically halting the request. It is resetting
the database connection partway through retrieving the results of the stored
proc. IIS attempts to close the connection for some reason, SQL ignores and
keeps pumping back data, then IIS raises a reset. MS confirmed that there
is no packet loss. MS has not determined why this is happening and has not
advised me to capture any additional data.
If someone has any ideas, I am willing to provide as much detail as
possible. I wanted to keep this initial post as concise as possible.
Thanks for any assistance you can provide,
-joe
EXCEPTION:
System.Data.SqlClient.SqlException: General network error. Check your
network documentation.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception,
TdsParserState state)
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException
exception, TdsParserState state)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
at System.Data.SqlClient.TdsParser.ReadNetlib(Int32 bytesExpected)
at System.Data.SqlClient.TdsParser.ReadBuffer()
at System.Data.SqlClient.TdsParser.ReadByteArray(Byte[] buff, Int32
offset, Int32 len)
at System.Data.SqlClient.TdsParser.ReadEncodingChar(Int32 length,
Encoding encoding)
at System.Data.SqlClient.TdsParser.ReadSqlValue(_SqlMetaData md, Int32
length)
at System.Data.SqlClient.TdsParser.ProcessRow(_SqlMetaData[] columns,
Object[] buffer, Int32[] map, Boolean useSQLTypes)
at System.Data.SqlClient.SqlDataReader.PrepareSQLRecord(Int32 i)
at System.Data.SqlClient.SqlDataReader.GetSqlString(Int32 i)
at System.Data.SqlClient.SqlDataReader.GetString(Int32 i)Have you eliminated hardware or installed software issues by trying to run
this on a different pair of machines?
HTH,
Kevin Spencer
.Net Developer
Microsoft MVP
Big things are made up
of lots of little things.
"Joe Ross" <joeross252@.hotmail.com> wrote in message
news:u55gTVJ$DHA.1796@.TK2MSFTNGP12.phx.gbl...
> I've been working with Microsoft support for over 3 weeks now on an
> intermittent General Network Error we're seeing in our production
> environment between our ASP.NET application and SQL Server 2000. They are
> continuing to work on the issue, but it seems as if our progress is
grinding
> to a halt. I asked this question on the newsgroups before going to MS but
> figured I'd give it another shot now that I have more information.
> The problem occurs intermittently (maybe 10 to 15 times per day). It
seems
> to be related to queries that return "large" amounts of data from SQL.
The
> problem started after we made an equipment change. We went from a single
> server hosting both our ASP.NET application and SQL Server 2000, to one
> server hosting IIS and another machine running SQL. An example of the
> exception thrown appears at the bottom of this post.
> MS had us perform 3 data captures initially: MPSRPT_MDAC on the IIS
machine,
> MPSRPT_MDAC on the SQL machine, and SQLDIAG on the database server. The
MS
> rep found nothing too out of the ordinary with our configuration, so he
> asked me to do a network capture on both machines while the error
occurred.
> I was able to get a good capture in a small window and sent it off to MS.
> The verdict is that IIS is basically halting the request. It is resetting
> the database connection partway through retrieving the results of the
stored
> proc. IIS attempts to close the connection for some reason, SQL ignores
and
> keeps pumping back data, then IIS raises a reset. MS confirmed that there
> is no packet loss. MS has not determined why this is happening and has
not
> advised me to capture any additional data.
> If someone has any ideas, I am willing to provide as much detail as
> possible. I wanted to keep this initial post as concise as possible.
> Thanks for any assistance you can provide,
> -joe
> EXCEPTION:
> System.Data.SqlClient.SqlException: General network error. Check your
> network documentation.
> at System.Data.SqlClient.SqlConnection.OnError(SqlException exception,
> TdsParserState state)
> at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException
> exception, TdsParserState state)
> at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
> at System.Data.SqlClient.TdsParser.ReadNetlib(Int32 bytesExpected)
> at System.Data.SqlClient.TdsParser.ReadBuffer()
> at System.Data.SqlClient.TdsParser.ReadByteArray(Byte[] buff, Int32
> offset, Int32 len)
> at System.Data.SqlClient.TdsParser.ReadEncodingChar(Int32 length,
> Encoding encoding)
> at System.Data.SqlClient.TdsParser.ReadSqlValue(_SqlMetaData md, Int32
> length)
> at System.Data.SqlClient.TdsParser.ProcessRow(_SqlMetaData[] column
s,
> Object[] buffer, Int32[] map, Boolean useSQLTypes)
> at System.Data.SqlClient.SqlDataReader.PrepareSQLRecord(Int32 i)
> at System.Data.SqlClient.SqlDataReader.GetSqlString(Int32 i)
> at System.Data.SqlClient.SqlDataReader.GetString(Int32 i)
>|||Unforunately, we're a small company and don't have the "luxury" of
duplicating the environment. However, there is plenty of horsepower there
to meet our needs. The web server is a Dell PowerEdge 1750 and the database
server is a Dell PowerEdge 2650 hooked into a PowerVault 220. Usage is
quite light.
I would love to be able to do what you say and I appreciate the value it
would bring to solving the problem, unfortunately, it's not realistic for us
at this time.
Thanks
-joe
"Kevin Spencer" <kevin@.takempis.com> wrote in message
news:OVDsMqJ$DHA.2524@.tk2msftngp13.phx.gbl...
> Have you eliminated hardware or installed software issues by trying to run
> this on a different pair of machines?
> --
> HTH,
> Kevin Spencer
> .Net Developer
> Microsoft MVP
> Big things are made up
> of lots of little things.
> "Joe Ross" <joeross252@.hotmail.com> wrote in message
> news:u55gTVJ$DHA.1796@.TK2MSFTNGP12.phx.gbl...
are
> grinding
but
> seems
> The
single
> machine,
> MS
> occurred.
MS.
resetting
> stored
> and
there
> not
exception,
Int32
>|||If it does turn out to be a hardware or installed software issue, I believe
you will have spent more than the cost of 2 machines in salaries trying to
find it.
HTH,
Kevin Spencer
.Net Developer
Microsoft MVP
Big things are made up
of lots of little things.
"Joe Ross" <joeross252@.hotmail.com> wrote in message
news:O1JOAaL$DHA.1732@.TK2MSFTNGP12.phx.gbl...
> Unforunately, we're a small company and don't have the "luxury" of
> duplicating the environment. However, there is plenty of horsepower there
> to meet our needs. The web server is a Dell PowerEdge 1750 and the
database
> server is a Dell PowerEdge 2650 hooked into a PowerVault 220. Usage is
> quite light.
> I would love to be able to do what you say and I appreciate the value it
> would bring to solving the problem, unfortunately, it's not realistic for
us
> at this time.
> Thanks
> -joe
>
> "Kevin Spencer" <kevin@.takempis.com> wrote in message
> news:OVDsMqJ$DHA.2524@.tk2msftngp13.phx.gbl...
run
> are
> but
> single
one
The
he
> MS.
> resetting
ignores
> there
has
> exception,
> Int32
columns,
>|||I respectfully disagree. Thank you for your overly helpful advice.
-joe
"Kevin Spencer" <kevin@.takempis.com> wrote in message
news:uW6oBBT$DHA.2476@.TK2MSFTNGP12.phx.gbl...
> If it does turn out to be a hardware or installed software issue, I
believe
> you will have spent more than the cost of 2 machines in salaries trying to
> find it.
> --
> HTH,
> Kevin Spencer
> .Net Developer
> Microsoft MVP
> Big things are made up
> of lots of little things.
> "Joe Ross" <joeross252@.hotmail.com> wrote in message
> news:O1JOAaL$DHA.1732@.TK2MSFTNGP12.phx.gbl...
there
> database
for
> us
> run
They
MS
It
SQL.
> one
the
> The
> he
to
the
> ignores
> has
possible.
your
System.Data.SqlClient.SqlInternalConnection.OnError(SqlException
bytesExpected)
Int32
> columns,
>|||> I respectfully disagree. Thank you for your overly helpful advice.
You stated originally that you had already spent 3 weeks on this problem
without a resolution. I don't know what your salaries are like, but 3 weeks
is a lot of man-hours, and Microsoft support is expensive. Maybe you should
disagree later, when and if you get it fixed.
Thank you for your overly defensive reply.
HTH,
Kevin Spencer
.Net Developer
Microsoft MVP
Big things are made up
of lots of little things.
"Joe Ross" <joeross252@.hotmail.com> wrote in message
news:OaXit3T$DHA.320@.TK2MSFTNGP10.phx.gbl...
> I respectfully disagree. Thank you for your overly helpful advice.
> -joe
> "Kevin Spencer" <kevin@.takempis.com> wrote in message
> news:uW6oBBT$DHA.2476@.TK2MSFTNGP12.phx.gbl...
> believe
to
> there
is
it
> for
to
an
> They
is
to
> MS
> It
> SQL.
to
> the
IIS
server.
so
> to
> the
that
and
as
> possible.
> your
> System.Data.SqlClient.SqlInternalConnection.OnError(SqlException
> bytesExpected)
> Int32
length,
md,
i)
>|||Joe -
At the risk of being 'overly' helpful, I would suggest that you may be
making too many assumptions about the operational data, for example the MS
assertion that no packets are being lost. The monitoring tools are not
infallible and they may in fact be wrong ( it even happened to me once long
ago ;-).
It sounds like MS is looking primarily at configuration data and not
performance data. Interaction under load between IIS and .NET may be
creating contention for resources or devices. If the queries are returning
'large' amounts of data, then the workload is not light, it's heavy and
there may well be a performance bottleneck in the server or network. It
could even be caused by a bad driver for a network device. Find every error
log and run every monitor on the system. There may be a lower level problem
within the apparent problem.
I'd also look very closely at application level performance, particularly
the heavy hitters, to see if you can reduce sudden bursts of demand. If the
heavy hitters are not critical, slow them down and spread out the demand.
If your delivery date is in extreme unction, you may even need to redesign
the queries to work around it.
In any case, I think you may need to fully instrument your server, collect
all the configuration and operation data available and do a more complete
analysis of your system. I've run across problems with complex interactions
in the past and a strictly analytic approach was the only way out.
For what it's worth ...
- Bill Breitmayer
"Joe Ross" <joeross252@.hotmail.com> wrote in message
news:OaXit3T$DHA.320@.TK2MSFTNGP10.phx.gbl...
> I respectfully disagree. Thank you for your overly helpful advice.
> -joe
> "Kevin Spencer" <kevin@.takempis.com> wrote in message
> news:uW6oBBT$DHA.2476@.TK2MSFTNGP12.phx.gbl...
> believe
to
> there
is
it
> for
to
an
> They
is
to
> MS
> It
> SQL.
to
> the
IIS
server.
so
> to
> the
that
and
as
> possible.
> your
> System.Data.SqlClient.SqlInternalConnection.OnError(SqlException
> bytesExpected)
> Int32
length,
md,
i)
>|||Bill-
I appreciate you taking the time to make such a detailed response. We are
going to take your advice and look at better instrumenting our application
to collect more meaningful data. We've realized that although we'll keep
working with MS on the issue, the solution will probably come from within.
I'll respond back to these groups when we make progress.
Thanks
-joe
"Bill Breitmayer" <billbreitmayer@.worldnet.att.net> wrote in message
news:pM60c.113110$hR.2187923@.bgtnsc05-news.ops.worldnet.att.net...
> Joe -
> At the risk of being 'overly' helpful, I would suggest that you may be
> making too many assumptions about the operational data, for example the MS
> assertion that no packets are being lost. The monitoring tools are not
> infallible and they may in fact be wrong ( it even happened to me once
long
> ago ;-).
> It sounds like MS is looking primarily at configuration data and not
> performance data. Interaction under load between IIS and .NET may be
> creating contention for resources or devices. If the queries are
returning
> 'large' amounts of data, then the workload is not light, it's heavy and
> there may well be a performance bottleneck in the server or network. It
> could even be caused by a bad driver for a network device. Find every
error
> log and run every monitor on the system. There may be a lower level
problem
> within the apparent problem.
> I'd also look very closely at application level performance, particularly
> the heavy hitters, to see if you can reduce sudden bursts of demand. If
the
> heavy hitters are not critical, slow them down and spread out the demand.
> If your delivery date is in extreme unction, you may even need to redesign
> the queries to work around it.
> In any case, I think you may need to fully instrument your server, collect
> all the configuration and operation data available and do a more complete
> analysis of your system. I've run across problems with complex
interactions
> in the past and a strictly analytic approach was the only way out.
> For what it's worth ...
> - Bill Breitmayer
>
> "Joe Ross" <joeross252@.hotmail.com> wrote in message
> news:OaXit3T$DHA.320@.TK2MSFTNGP10.phx.gbl...
trying
> to
> is
value
> it
realistic
trying
> to
> an
production
> is
> to
information.
day).
from
a
> to
of
> IIS
> server.
configuration,
> so
error
off
of
> that
> and
> as
Check
> length,
> md,
> i)
>