Make sure the reverse proxy you are using for SSL termination is setting the X-Scheme header to https so OctoPrint knows it is being accessed through HTTPS and not HTTP and knows to generate the correct URLs.
Example: If you've set up your reverse proxy to serve OctoPrint on its address under https://<your server> you'll need to make it set the header X-Scheme: https in its requests to OctoPrint.
There are examples on the wiki on how to set everything up correctly. Alternatively, you can force OctoPrint to always assume HTTPS by editing the configuration file.