Access Forbidden?

So, new development:

After starting in safe mode, I have no issues. So I'm going to assume one of my plugins (Looking at you, TouchUI) is having issues. I'm going to disable TouchUI first. If that doesn't work, I'll hit OctoLapse, since that was giving me drek, too. After that, I'm rolling dice.

Don't forget to take this to the author of that plugin. I'm sure they'll be interested in your story. There may be something that's related to a combination of your settings plus the plugin or perhaps the timing of events that led up to this.

Babysit your logs, though. You still have trouble from your fallout security geek.

So new, new development, Octolapse AND TouchUI are broken. TouchUI was stopping me from saving configs, but Octolapse has brought to light some other issues. I'll be taking this over to them, but apparently I cannot update any printer profiles, and OctoLapse shows them as "not set up."

Write a pip script, store it online, ssh in, wget said script, run it, automagically install 15 plugins at once, reboot, win.

That also doesn't solve my Twin Instance config issue.

oh, I thought you'd narrowed it down to a broken plugin?

It is a broken plugin. OctoLapse and TouchUI. I'm talking with the author of OctoLapse now, trying to figure out what the problem actually is, since he's never seen it before.

That is a good idea, though.

I haven't marked this as solved, since I want to ACTUALLY solve the issue, first. Then I'll document it here, and mark it as the answer.

To me, that sounds like there's a JavaScript error that's being thrown. Running the Developer's Console in your browser (F12 in Windows or right-mouse click -> Inspect in OSX) should reveal some light on the problem there. The author would want to know this of course.

Try temporarily disabling both plugins and see if you can then do your Settings update.

I did state that uninstalling both gave me the ability to save again... I think.

Just now found this while searching for possible causes for the problems @Xlaits is reporting. I wanted to add some info here in case others have ideas.

First, from the logs I've seen, OP is under active attack. There are strange errors that I've never seen before like this from the javascript console:

TypeError: OctoPrint.browser.logout(...).done(...).error is not a function[Learn More] packed_core.js:2406:18
LoginStateViewModel/self.logout
https://xlaitsnet.tk/ender/static/webassets/packed_core.js:2406:18
init/</<
https://xlaitsnet.tk/ender/static/webassets/packed_libs.js:11395:441
dispatch
https://xlaitsnet.tk/ender/static/webassets/packed_libs.js:5183:16
add/elemData.handle
https://xlaitsnet.tk/ender/static/webassets/packed_libs.js:4992:6

There are some Octolapse errors too, but they are caused by an ajax request that's being blocked (403 in the load settings routine).

Edit: Accidentally posted early. Adding more.

I think the only way to get to the bottom of this would be to disable all plugins but Octolapse and start looking into issues one at a time. Preferably OP could try a fresh install (who knows what those attackers could be doing) and only add Octolapse. I strongly suspect this would solve all of these problems.

Also, @Xlaits, what do you mean by 'twin instances' exactly?

Active probing looking for attack vectors :wink: Untargeted though.

Just so that no one is hunting down the wrong trail here: I've actually seen that - my error, fixed in 1.3.10rc1+ by @BillyBlaze. Only triggered on logout though, so shouldn't interfere with regular operation.

Thanks for chiming in @foosel! I also see this in the log file:

2018-11-27 09:41:57,149 - tornado.access - WARNING - 403 POST /plugin/octolapse/loadSettings (::ffff:108.162.210.191) 953.05ms
2018-11-27 09:41:58,055 - tornado.access - WARNING - 403 GET /api/system/commands (::ffff:108.162.210.191) 11.27ms
2018-11-27 09:48:49,835 - tornado.access - WARNING - 403 GET /api/languages (::ffff:108.162.212.96) 11.18ms
2018-11-27 09:48:49,850 - tornado.access - WARNING - 403 GET /plugin/logging/ (::ffff:108.162.212.96) 10.75ms
2018-11-27 09:48:52,404 - tornado.access - WARNING - 403 POST /api/settings (::ffff:108.162.212.96) 37.82ms
2018-11-27 09:48:53,502 - tornado.access - WARNING - 403 POST /api/connection (::ffff:108.162.212.96) 14.20ms
2018-11-27 09:49:06,025 - tornado.access - WARNING - 403 GET /api/languages (::ffff:108.162.212.96) 23.67ms
2018-11-27 09:49:06,063 - tornado.access - WARNING - 403 GET /plugin/logging/ (::ffff:108.162.212.96) 28.43ms
2018-11-27 09:49:16,491 - tornado.access - WARNING - 403 POST /api/settings (::ffff:108.162.212.96) 38.15ms
2018-11-27 09:49:17,766 - tornado.access - WARNING - 403 POST /plugin/touchui/css (::ffff:108.162.212.96) 177.54ms

The first 403 there (/plugin/octolapse/loadSettings) will certainly prevent Octolapse from working. Any ideas what could be causing this?

The only idea that I have is that the session got lost somehow. The question is why though. UI hiccups usually get cured by logging out and back in / cleaning the browser session. But apparently that didn't help here.

I wonder how OctoPrint would behave in the presence of a cookie muncher of some kind? I don't use them but ghostery.com or similar. Or perhaps there's a browser setting that's wrong. Or perhaps we're looking at something like...

  1. browser resolves octopi.local as an IPv4 address
  2. session is created under that
  3. for whatever reasons arp/dns decides that IPv6 is also available
  4. the browser is now using that
  5. the browser can't see the earlier session...

I dunno. But I do know that local storage in a browser is painfully stupid like this. I might be surfing both http://localhost/ and http://10.20.30.240/ (both on my MacBook) and it will create two separate local storage databases and it behaves as if I'd lost my session. And I also know that DNS lookups will sometimes provide either IPv4 and IPv6 versions based upon the direction of the wind, presumably.

Another plot twist:

  1. user starts session
  2. user walks away
  3. laptop goes into sleep mode
  4. upon being awoken, the IPv4/IPv6 switchover takes place
  5. reload page, lost session

I have two instances of OctoPrint running, since I need to run two printers at the same time. One instance worked fine, but the MAIN instance was the broken one.

I have the backup img sort of ready, I just need to reflash it, then create a test admin account for you to mess with, then can it up again.

How are the instances running? Are they both behind haproxy? Are you running each from their own separate base directory or trying to run them both from the same base dir? Since this seems to be a rather non-standard setup, you might have to give us more details on how it's set up. It could be that the 2 instances are fighting with each other, or your browser is unable to distinguish between them.

They have their own base directories, but use a common daemon folder. I'm using HAproxy to separate the webpages as well.

I posted a link to the image of my current OctoPi setup here. It is also where @FormerLurker and I are working to solve this... unique problem.

I seem to attract unique problems...

@Xlaits, we all have our own special problems is seems, lol! My printer upgrade just arrived after over a year of waiting, so it may take me a while before I can take a look. If anyone else wants to take a stab at it in the meanwhile I say go for it!

I will be re-uploading a new image with a linux-side test admin account at some point, when I have some time to do so.