BootFailed Cannot Connect To Database

[Emetico]

Which Umbraco version are you using? (Please write the exact version, example: 10.1.0)

7.15.11, 8.18.15, 10.8.7, 13.5.2

Bug summary

We are experiencing issues with our Umbraco solutions hosted at Curanet, where the applications lose connection to the database after the server (either web or SQL) has been restarted. This occurs because IIS starts before SQL Server is ready, resulting in stale connections that are not refreshed. Consequently, this leads to a BootFailed exception when attempting to start the application. The loss of connection is sporadic; it does not happen every time and can affect different solutions at different times. Our Umbraco solutions are running the following versions:

• Umbraco 7.15.11
• Umbraco 8.18.15
• Umbraco 10.8.7
• Umbraco 13.5.2

Specifics

• Connection Loss Timing: The database connection is lost after a server restart, particularly when IIS initializes before SQL Server.
• BootFailed Exception: When the connection issue arises, we receive a BootFailed exception, indicating that the application cannot establish a valid connection to the database due to stale connections.
• Environment: All solutions are hosted with Curanet, and the problem is sporadic across different Umbraco versions.

Steps to reproduce

1. Environment Setup:

• Ensure the Umbraco solutions are hosted on a Curanet server with the following versions:
  Umbraco 7.15.11
  Umbraco 8.18.15
  Umbraco 10.8.7
  Umbraco 13.5.2

2. Restart the SQL Server:

• Restart the SQL Server instance hosting the database for the Umbraco solution.

3. Restart the Web Server:

• Restart the web server (IIS) hosting the Umbraco application.

4. Monitor Startup Sequence:

• Observe the startup sequence to ensure that the web server (IIS) starts before the SQL Server is fully operational.

5. Check Application Logs:

• After the web server starts, monitor the Umbraco application logs for any entries indicating a BootFailed exception or database connection errors.

6. Verify Connection Status:

• Attempt to access the Umbraco backoffice or any front-end page.
• Confirm that the application fails to establish a connection to the database, resulting in a BootFailed error.

Expected result / actual result

Expected Result:

• Upon restarting both the SQL Server and the web server (IIS), the Umbraco application should successfully connect to the database without any errors.
• The application should initialize correctly, allowing access to the backoffice and front-end without encountering a BootFailed exception.

Actual Result:

• After restarting the SQL Server and web server, the Umbraco application fails to connect to the database due to stale connections.
• The application logs show a BootFailed exception, indicating that the application could not establish a valid connection to the database.
• Users are unable to access the backoffice or any front-end pages, resulting in service disruption.

Proposed Solution:

To address the database connection issues leading to the BootFailed exception, we suggest the following enhancements for Umbraco:

1. Development of BootFailedNotification:

• Implement a public UmbracoBootFailedNotification(BootFailedReason reason) as outlined in the Umbraco notifications documentation.
• This would allow users to check if the reason for the boot failure is related to database connectivity (e.g., CannotConnectToDatabase).
• By detecting this specific failure, applications could use IHostApplicationLifetime.StopApplication() to gracefully stop the application until the database connection is restored.

2. Utilization of ClearPool():

• Consider incorporating ClearPool() into Umbraco's error handling for database connections if feasible.
• This could help in clearing stale connections and improving the application's ability to recover from temporary connection issues.

[github-actions]

Hi there @Emetico!

Firstly, a big thank you for raising this issue. Every piece of feedback we receive helps us to make Umbraco better.

We really appreciate your patience while we wait for our team to have a look at this but we wanted to let you know that we see this and share with you the plan for what comes next.

• We'll assess whether this issue relates to something that has already been fixed in a later version of the release that it has been raised for.
• If it's a bug, is it related to a release that we are actively supporting or is it related to a release that's in the end-of-life or security-only phase?
• We'll replicate the issue to ensure that the problem is as described.
• We'll decide whether the behavior is an issue or if the behavior is intended.

We wish we could work with everyone directly and assess your issue immediately but we're in the fortunate position of having lots of contributions to work with and only a few humans who are able to do it. We are making progress though and in the meantime, we will keep you in the loop and let you know when we have any questions.

Thanks, from your friendly Umbraco GitHub bot 🤖 🙂

[Hvid]

In my opinion, the BootFailedException should not be the last stop. Is there a reason why the boot is not retried?

[Emetico]

In my opinion, the BootFailedException should not be the last stop. Is there a reason why the boot is not retried?

From what I can see from the logs files, it tries to boot around 5 times before it dies. But each time it uses a stale connection that are not refreshed.

[geoffbeaumont]

@Emetico What log level do you have Umbraco set to? When I see this on Umbraco 13 I don't get anything other than the usual start-up messages in the log (set to log level Error - and this would definitely seem to be a major error!). Just get an application is not available message in browsers.

Nothing else I work with has this issue with permanently failing if the database is unavailable on start-up, and it's a major (and long standing) problem. An unavailable database is a highly predictable event in many hosting environments, especially if everything is being rebooted following a Windows update - there's no way it should put Umbraco into a permanently unavailable state.