Since Blender Version 2.83, there is the Adaptive Sampling feature, which allows Blender to use different sampling rates in the same image.
That means more noisy parts of the Image that require more samples get more samples, but Blender doesn’t waste time in parts of the image which doesn’t require as many samples. Which can make it faster multiple times but also not faster at all. It hardly depends on the Image.
Just check the Box at Adaptive Sampling and you are done. You could also add value for minimum samples or mess with the Noise Threshold but I just leave it how it is.