How to check if server and port settings are correctly configured?

[colin-sim]

This is a followup to #1073.

When connecting to a database, a MySqlException exception with MySqlErrorCode.UnableToConnectToHost error code is raised in the case:

• where the server and port are misconfigured.
• where the server is no longer accepting new connections.

I would like to setup a retry policy in the case where the database server is under pressure but just error when the server and/or port is misconfigured. Is there a way to do that in MySqlConnector?

[bgrainger]

where the server is no longer accepting new connections

I would expect to see MySqlErrorCode.ConnectionCountError in that scenario.

If they're both reporting MySqlErrorCode.UnableToConnectToHost, I'm not sure if there's a good way to distinguish these error cases.

[colin-sim]

Hi @bgrainger.

After some more experimentation, I have observed:

1. MySqlErrorCode.ConnectionCountError is thrown when I run my test application in the debugger (Visual Studio 2019).
2. The exact application throws MySqlErrorCode.UnableToConnectToHost when run from the console.
3. When a back off policy is applied, I see MySqlErrorCode.UnableToConnectToHost thrown initially then MySqlErrorCode.ConnectionCountError.

Is there a timing/race issue on the client side?

[bgrainger]

Can you give full exception call stacks for the three cases?

when I run my test application in the debugger

Is that a first-chance exception, or the final unhandled exception?

[colin-sim]

Hi @bgrainger. Below are the stack traces for each of the above scenarios.

Scenario 1
Not first chance exception but the one caught at the top level try-catch block.

Error code: ConnectionCountError. Description: Too many connections
    at MySqlConnector.Core.ServerSession.ReceiveReplyAsyncAwaited(ValueTask\`1 task) in /_/src/MySqlConnector/Core/ServerSession.cs:line 908
    at MySqlConnector.Core.ServerSession.ConnectAsync(ConnectionSettings cs, MySqlConnection connection, Int32 startTickCount, ILoadBalancer loadBalancer, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 440
    at MySqlConnector.Core.ConnectionPool.ConnectSessionAsync(MySqlConnection connection, String logMessage, Int32 startTickCount, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 363
    at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, Int32 startTickCount, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 94
    at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, Int32 startTickCount, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 124
    at MySqlConnector.MySqlConnection.CreateSessionAsync(ConnectionPool pool, Int32 startTickCount, Nullable\`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 911
    at MySqlConnector.MySqlConnection.OpenAsync(Nullable\`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 405
    at MySql.Program.ExecuteQueryAsync(String connectionString, UInt32 timeout) in S:\\Muck\\MySql\\Program.cs:line 93
    at MySql.Program.ExecuteQueryAsync(String connectionString, UInt32 timeout) in S:\\Muck\\MySql\\Program.cs:line 94

Scenario 2

Error code: UnableToConnectToHost. Description: Couldn't connect to server
    at MySqlConnector.Core.ServerSession.ConnectAsync(ConnectionSettings cs, MySqlConnection connection, Int32 startTickCount, ILoadBalancer loadBalancer, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 552
    at MySqlConnector.Core.ConnectionPool.ConnectSessionAsync(MySqlConnection connection, String logMessage, Int32 startTickCount, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 422
    at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, Int32 startTickCount, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 126
    at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, Int32 startTickCount, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 126
    at MySqlConnector.MySqlConnection.CreateSessionAsync(ConnectionPool pool, Int32 startTickCount, Nullable\`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 940
    at MySqlConnector.MySqlConnection.OpenAsync(Nullable\`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 445
    at MySql.Program.ExecuteQueryAsync(String connectionString, UInt32 timeout) in S:\Muck\MySql\Program.cs:line 93
    at MySql.Program.ExecuteQueryAsync(String connectionString, UInt32 timeout) in S:\Muck\MySql\Program.cs:line 94

Error code: UnableToConnectToHost. Description: Unable to connect to any of the specified MySQL hosts.
    at MySqlConnector.Core.ServerSession.ConnectAsync(ConnectionSettings cs, MySqlConnection connection, Int32 startTickCount, ILoadBalancer loadBalancer, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 432
    at MySqlConnector.Core.ConnectionPool.ConnectSessionAsync(MySqlConnection connection, String logMessage, Int32 startTickCount, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 422
    at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, Int32 startTickCount, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 126
    at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, Int32 startTickCount, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 126
    at MySqlConnector.MySqlConnection.CreateSessionAsync(ConnectionPool pool, Int32 startTickCount, Nullable\`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 940
    at MySqlConnector.MySqlConnection.OpenAsync(Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 445
    at MySql.Program.ExecuteQueryAsync(String connectionString, UInt32 timeout) in S:\Muck\MySql\Program.cs:line 93
    at MySql.Program.ExecuteQueryAsync(String connectionString, UInt32 timeout) in S:\Muck\MySql\Program.cs:line 94

Scenario 3

Error code: UnableToConnectToHost. Description: Unable to connect to any of the specified MySQL hosts.
    at MySqlConnector.Core.ServerSession.ConnectAsync(ConnectionSettings cs, MySqlConnection connection, Int32 startTickCount, ILoadBalancer loadBalancer, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 432
    at MySqlConnector.Core.ConnectionPool.ConnectSessionAsync(MySqlConnection connection, String logMessage, Int32 startTickCount, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 422
    at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, Int32 startTickCount, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 126
    at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, Int32 startTickCount, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 126
    at MySqlConnector.MySqlConnection.CreateSessionAsync(ConnectionPool pool, Int32 startTickCount, Nullable\`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 940
    at MySqlConnector.MySqlConnection.OpenAsync(Nullable\`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 445
    at MySql.Program.ExecuteQueryAsync(String connectionString, UInt32 timeout) in S:\Muck\MySql\Program.cs:line 90
    at MySql.Program.ExecuteQueryAsync(String connectionString, UInt32 timeout) in S:\Muck\MySql\Program.cs:line 90

Error code: ConnectionCountError. Description: Too many connections
    at MySqlConnector.Core.ServerSession.ReceiveReplyAsyncAwaited(ValueTask\`1 task) in /_/src/MySqlConnector/Core/ServerSession.cs:line 910
    at MySqlConnector.Core.ServerSession.ConnectAsync(ConnectionSettings cs, MySqlConnection connection, Int32 startTickCount, ILoadBalancer loadBalancer, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 440
    at MySqlConnector.Core.ConnectionPool.ConnectSessionAsync(MySqlConnection connection, String logMessage, Int32 startTickCount, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 422
    at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, Int32 startTickCount, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 126
    at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, Int32 startTickCount, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 126
    at MySqlConnector.MySqlConnection.CreateSessionAsync(ConnectionPool pool, Int32 startTickCount, Nullable\`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 940
    at MySqlConnector.MySqlConnection.OpenAsync(Nullable\`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 445
    at MySql.Program.ExecuteQueryAsync(String connectionString, UInt32 timeout) in S:\Muck\MySql\Program.cs:line 90
    at MySql.Program.ExecuteQueryAsync(String connectionString, UInt32 timeout) in S:\Muck\MySql\Program.cs:line 90

[bgrainger]

The following call stack will happen any time the client can open a connection to the server, but the server returns an error packet:

Error code: ConnectionCountError. Description: Too many connections
    at MySqlConnector.Core.ServerSession.ReceiveReplyAsyncAwaited(ValueTask\`1 task) in /_/src/MySqlConnector/Core/ServerSession.cs:line 910

If that doesn't happen, then the most likely cause is that a connection to the server couldn't actually be established. I don't see evidence of a bug/race condition yet.

Are you claiming that you get different (but completely consistent and reproducible) exceptions depending on whether you run the program under the debugger or from the console? (And it's the same Debug or Release build in both cases?)

[colin-sim]

Are you claiming that you get different (but completely consistent and reproducible) exceptions depending on whether you run the program under the debugger or from the console?
Yes.

And it's the same Debug or Release build in both cases?
Yes

When the connection couldn't be established, could a different error be returned so the client can distinguish between this case and the case where the server/port is misconfigured? E.g. HostIsBlocked when connection cannot be established and UnableToConnectToHost when server/port is misconfigured.

[bgrainger]

could a different error be returned

In general, no: that would be a breaking change for any code currently examining the current errors.

When the connection couldn't be established, could a different error be returned so the client can distinguish between this case and the case where the server/port is misconfigured?

I don't yet understand how those two scenarios are different; can you elaborate?

"establishing a connection" is performing the TCP 3-way handshake (https://developer.mozilla.org/en-US/docs/Glossary/TCP_handshake) followed by reading the "server hello" packet the server sends, and responding with a (hashed) password. If there are too many connections, the server will send an error packet instead of the hello packet.

If the TCP connection can't be established, UnableToConnectToHost is thrown. If it is established, but the server sends an error packet, the error (usually ConnectionCountError) is thrown.

If you think you're observing something different, you could collect a packet capture (with Wireshark, tcpdump, etc.) and show how MySqlConnector isn't returning the right response.

If your use case for your application is that server/port might commonly be incorrect (is this user-entered data that needs to be verified?) you might need to take more control by opening a Socket yourself and making a low-level connection. Then you could distinguish between OpenAsync timing out, a good or bad handshake being returned, and any other error condition. If the connection looked "good", you could close the Socket then connect with MySqlConnector.

[colin-sim]

HI @bgrainger. Is there any way I can share the test app and environment (EC2 instance) with you? It may be easier for you to understand the heuristic, just in case I'm not explaining the scenarios properly.

What I'm observing is that UnableToConnectToHost is raised when the server or possibly the client is under load. Given that when the app is raising the expected ConnectionCountError when run in debug mode this makes me think it's more likely the error is due to an issue when client is under load.

[bgrainger]

Sure, you can email details to (my GitHub username) @gmail.com.

[colin-sim]

@bgrainger was unable to reproduce the error on his environment, which inspired me to repeat my checks in a sanitized environment. Sure enough the issue isn't present there so we can conclude that the issue described above is unique to my machine (either a resourcing issue or contention with security software).

Thanks for all your help @bgrainger!