After some time away on other systems, I have had a project purchased for a customer running a module from China from Quectel. It is a nice little module which supports MQTT however the solution that it runs was unable to connect to AWS IoT Core for love or money. I love AWS and cloud instance of MQTT brokers in general, but the problem with it not being your own server is that these types of issues are extremely hard to debug. At first I thought the Quectel module wasn't processing/presenting the certificates and/key properly or it was strongly to validate the server's certificate correctly. To resolve the first two is easy and so I set up a small Linux server and deployed a secure instance of Mosquitto there. No issue, the unit was able to connect perfectly and the certificates from AWS were exactly as I expected them to be. Likewise, my own server's certificate was able to be validated by the device and so there didn't seem to be an issue here.
The certificates were confirmed to be valid and matched with the keys we were sending and yet still there was no way to connect to IoT Core and nothing in the logs to tell us what the issue was or even to show that they were reaching the correct server. As always with these types of issues it boils down to a small line hidden away in the documentation. With all the experience in the team no one had ever seen the issue before and so wasted almost a week of engineering time tracking it down.
What was the solution? https://github.com/brentonjudge/StoreLocator/issues/1
Lessons learned are always a companies most valued resource.