It's difficult to say what may be going on. I recommend looking in your browser console to see if any error messages are being displayed. Is it possible it's losing the SignalR connection and is needing to reconnect before loading the page?

No error in console, the strange thing is that this behavior also happens on the first click when the application has just loaded. 

Without any code to review, it's nearly impossible to know what is causing the issue. It could be so many things, like is there a loop or Task.Delay being used when the app or page initializes, or is there maybe an OnAfterRender or OnAfterRenderAsync method being used without making sure it's the first render?

Usually just use this method to load the components (protected override async Task OnInitializedAsync()) something important I use MudBlazor as a framework

I've never used MudBlazor, so my best guess would be to maybe try removing one of their components at a time to narrow down which component may be causing the delay. If you can narrow it down to a specific component that isn't related to your code or hosting setup, you may be able to reach out to the vendor for support.

I have been watching the console and it gives me this error even when I am debugging Error: Connection disconnected with error Error: Server timeout elapsed without receiving a message from the server.    any ideas?

This seems to point back to my original thought. Blazor Server requires a constant SignalR connection in order to update the screen (referred to as re-painting the screen sometimes). When that connection breaks, it sounds like your app is hanging until the connection is re-established. Your server app pool may be recycling or something? Also, you might see a disconnected error in local development whenever you save changes in Visual Studio, because it has to restart the app.