Let's Encrypt server cert bootstrapping, scheduled 4 days ago
Helios, Sol, and Luna will implement automatic Let's Encrypt bootstrapping for HTTP, email, and FTP services tonight beginning at 11 PM EST (-0500 GMT). Bootstrapping only covers the primary server certificate, not individual sites, which still require a separate SSL addon service. No outage is anticipated beyond a 10-30 second outage for each service that is brought up with the new certificate. Server certificates will automatically renew every 80 days.
10:30 PM EST: maintenance began earlier than anticipated.
11 PM EST: maintenance window concluded.
29th November 2016
No incidents reported
28th November 2016
At approximately 6:05 PM EST, Sol experienced a persistent brute-force attack on multiple web resources from a small collection of IP addresses. Typically, these are picked up and blocked without affecting performance beyond a minor blip as the system identifies and responds to the threat.
A bug was uncovered during IP address matching (grep/egrep pattern distinction) that rendered automatic DoS protection ineffective. The issue has since been resolved and propagated to other servers. As of 6:20 PM EST, the system is considered stable.