Errors with embedded forms
Incident Report for Department of Better Technology
Postmortem

What happened

At approximately 13:00 ET on 6/19/2014 we pushed a deploy to production that introduced a bug that preventing users from submitting responses inside of Screendoor. This bug affected response forms that were embedded using our iframe embed code. At approximately 16:00 ET, we were alerted that some users were unable to submit responses. At 16:30 ET, we had resolved the issue on all affected servers.

The technical details

Our production servers are automatically configured to forward any requests made over the 'http' protocol to 'https', with the exception of embedded response forms, which are served over http to allow for custom stylesheets that are hosted on non-https domains. When the form attempted to save its values to the server, it requested the non-https url for the responses endpoint, which threw a redirect and canceled the form submission. Normally, our Q+A process would have caught this, but because our staging server was not configured to redirect non-https requests, we did not become aware of the error until it occurred in production.

How we fixed it

We pushed a fix to explicitly set the submission endpoint to the https URL when rendering a form inside of an embedded page.

Going forward

We have already reconfigured our staging servers to match the behavior of our production servers, so that in the future, this sort of bug would be caught during our normal Q+A process. It was not acceptible for this bug to make it into our production environment. We want to apologize for any issues we may have caused, and we'd like to offer our assurance that the reliability and stability of Screendoor is of the highest importance to us, and we're working every day to ensure that incidents like this one never happen again. We appreciate your trust in us and we’re going to live up to it.

Posted Jun 19, 2014 - 16:07 CDT

Resolved
At approximately 13:00 ET on 6/19/2014 we pushed a deploy to production that introduced a bug that preventing users from submitting responses inside of Screendoor. This bug affected response forms that were embedded using our iframe embed code. At approximately 16:00 ET, we were alerted that some users were unable to submit responses. At 16:30 ET, we had resolved the issue on all affected servers.
Posted Jun 19, 2014 - 16:05 CDT