All systems are operational

Past Incidents

4th December 2016

No incidents reported

3rd December 2016

No incidents reported

2nd December 2016

No incidents reported

1st December 2016

No incidents reported

30th November 2016

Let's Encrypt server cert bootstrapping, scheduled 4 days ago

Helios, Sol, and Luna will implement automatic Let's Encrypt bootstrapping for HTTP, email, and FTP services tonight beginning at 11 PM EST (-0500 GMT). Bootstrapping only covers the primary server certificate, not individual sites, which still require a separate SSL addon service. No outage is anticipated beyond a 10-30 second outage for each service that is brought up with the new certificate. Server certificates will automatically renew every 80 days.

10:30 PM EST: maintenance began earlier than anticipated.

11 PM EST: maintenance window concluded.

29th November 2016

No incidents reported

28th November 2016

Sol Sol DoS

At approximately 6:05 PM EST, Sol experienced a persistent brute-force attack on multiple web resources from a small collection of IP addresses. Typically, these are picked up and blocked without affecting performance beyond a minor blip as the system identifies and responds to the threat.

A bug was uncovered during IP address matching (grep/egrep pattern distinction) that rendered automatic DoS protection ineffective. The issue has since been resolved and propagated to other servers. As of 6:20 PM EST, the system is considered stable.