Status Updates | Hostineer

High rate packet loss

Status Updates | Hostineer — Mon, 14 Jul 2025 19:32:00 +0000

Presently seeing a high rate of packet loss on edge router, 205.214.78.36 within DataBank's Atlanta data center. Investigating.

Panel OS upgrade

Status Updates | Hostineer — Fri, 04 Apr 2025 03:05:00 +0000

cp.hostineer.com will be temporarily unavailable on Saturday, April 5 between 6 PM and 12 AM ET while an OS upgrade is underway. The total outage is expected to last no more than 30 minutes. During this time, use your shortcut panel access at https://DOMAIN/cpadmin to login to the specific server control panel.

Fiber Move

Status Updates | Hostineer — Tue, 26 Sep 2023 17:07:55 +0000

We have scheduled a brief network outage for this coming Saturday, September 30th at 1:00 AM EST. The total outage could last up to 30 minutes while we move the DIA fiber from 1100 White Street to 1150 White Street. During the outage period from 1:00 AM EST to 1:30 AM EST, you should expect high latency and/or a total outage. We apologize for any inconvenience this may cause and we will make every effort to keep the outage duration as short as possible. Thank you for your patience and understanding as we work to improve our DIA Network performance and redundancy.

Data Center Migration

Status Updates | Hostineer — Mon, 28 Aug 2023 23:04:41 +0000

Servers will be moved to an adjacent data center beginning on Saturday, September 3 between 12 - 3 AM ET (-0400 GMT). During this window, servers will be taken offline as they are physically transported next door. A second 3-hour window is scheduled for Sunday, September 4 at 12 AM ET. No other impact is expected with this move.

XFS corruption

Status Updates | Hostineer — Tue, 22 Sep 2020 15:51:00 +0000

11:51 AM EDT: investigating transient corruption issue reported on xfs filesystem. 12:12 PM: non-destructive recovery exhausted, zeroing transaction journal and performing recovery on filesystem. 12:25 PM: recovery successful. System operational.

Atlas outage

Status Updates | Hostineer — Wed, 05 Aug 2020 10:55:00 +0000

Atlas experienced a hard lockup at 6:55 AM EDT (-0400 GMT). The server has been rebooted manually and undergoing a mandatory filesystem check. **7:26 AM**: filesystem check completed. System stabilized. Will continue to monitor. **9:17 AM**: server flapping. Investigating. This is also affecting panel access as Atlas serves as the default panel endpoint. **9:35 AM**: memory raised on server, forced fsck disabled on boot.

Panel login outage

Status Updates | Hostineer — Tue, 07 Jan 2020 18:00:00 +0000

An outage was recorded today in the Login portal. This was caused by a refactoring of the Login application into modules, many of which were marked to be ignored by the application's .gitignore file.

ASR networking outages

Status Updates | Hostineer — Fri, 27 Dec 2019 13:15:00 +0000

Automated system restarts on Orion are causing the network, upon reboot, to remain inactive. This is under further review. 9:28 EST: NetworkManager has been disabled on Orion, relying on static assignment. Issue resolved.

Hardware failure

Status Updates | Hostineer — Sun, 06 Oct 2019 15:42:00 +0000

Hardware error on Helios. Investigating... **2:18 PM EDT (-0400 GMT)**: Issue resolved. A hard drive failure was experienced in Helios on a hardware RAID10 arrangement. RAID10 is built to be durable, able to sustain multiple drive failures without compromising system stability. For reasons still under further research, the drive failure created an anomaly on the RAID10-backed swap partition that in turn induced a kernel panic. **4:12 PM EDT**: subsystem dropped again. Appears to be a bad RAID controller. **5:30 PM EDT**: RAID controller replaced. Issue resolved.

Cabinet changes

Status Updates | Hostineer — Sun, 29 Sep 2019 16:37:17 +0000

Beginning tonight, September 29 at 10 PM EDT (-0500 GMT), servers and network will be subject to brief rolling outages as infrastructure is reorganized in the cabinet. Maintenance will conclude by 1 AM. Rolling outages will last no more than 10 minutes per outage, if any are incurred.

Atlas storage resize

Status Updates | Hostineer — Wed, 15 May 2019 17:28:17 +0000

Atlas will be taken down today, May 16 at 1 AM EDT (-0400 GMT) to resize storage. Two reboots will be necessary to properly resize the partition. Downtime will not exceed 10 minutes and should complete within 5 minutes.

Apache upgrades

Status Updates | Hostineer — Tue, 02 Apr 2019 17:13:35 +0000

Atlas, Luna, Sol, Helios will undergo Apache upgrades later today beginning at 12 AM EDT (-0400 GMT) to address [CVE-2019-0211](https://access.redhat.com/security/cve/cve-2019-0211), a severe privilege escalation. All four platforms use non-standardized distributions (Delta+ platforms do not) that require a manual build. Expect 5 - 30 minutes outage per server as updates are applied.

MySQL remote access restrictions

Status Updates | Hostineer — Thu, 28 Mar 2019 05:01:00 +0000

Due to an ever increasing rise in threats from compromised machines, all platforms before v6.5 (Luna) now disable remote MySQL. To continue to use remote MySQL access, open a ticket to request a migration to a newer platform.

SRS + rspamd upgrades

Status Updates | Hostineer — Mon, 24 Dec 2018 19:25:38 +0000

Sol and Luna will receive [SRS + rspamd](https://updates.hostineer.com/2018/12/mail-enhancements-on-atlas-delta/) support on January 12, 2019 beginning at approximately 12 PM EST (-0500 GMT). During this time minor interruptions may occur to SMTP/IMAP/POP3 services.

Periodic load spikes

Status Updates | Hostineer — Fri, 21 Dec 2018 17:24:00 +0000

Helios has been experiencing substantial load spikes every 24-48 hours. As this platform is nearing EOL (July 2019), it is being investigated to its fullest extent. Expect partial disruptions, including a kernel upgrade, over the next 24 hours as engineers work to isolate the cause.

Email enhancements

Status Updates | Hostineer — Sat, 15 Dec 2018 16:05:00 +0000

Atlas is undergoing a planned upgrade to expand email capabilities, including new spam filtering and before-queue milter. Periodic email disruption is expected over the next hour. **1:21 PM EST**: Maintenance window concluded. See [Mail Enhancements on Atlas + Delta](https://updates.hostineer.com/2018/12/mail-enhancements-on-atlas-delta/) for further details.

MySQL hard lock

Status Updates | Hostineer — Thu, 16 Aug 2018 06:30:00 +0000

A lock was noted on August 16 at approximately 3:18 AM EDT (-0400 GMT), which prevented access to MySQL. Internal monitoring is designed to catch these events and restart the affected service automatically. A further analysis of the event yielded a situation that could arise in which the MySQL server is active, but non-responsive to connections. A second check, which validates connection counts depends upon this first check, which if it hangs indefinitely during authentication handshake, results in a deadlock. A timeout has been added to both MySQL service checks, which will resolve such situations should they occur in the future.

Network outage

Status Updates | Hostineer — Wed, 13 Jun 2018 10:55:00 +0000

Network appears to be down at the moment within the data center. DC has been notified. Waiting on response. 8:59 AM EDT. Network is stabilized. Waiting on RFO upstream. 2018/06/13: RFO follows. > This message is to the address the network outage that occurred on 6/13/2018 at 6:34am After research and analysis the reason for this outage was directly linked to a final configuration that was not fully saved during our Monday June 4th network upgrade. The network device reverted back to a previous saved configuration resulting in the unexpected outage on 6/13. The new configuration was replaced and Fully saved/Verified to insure this type of outage cannot occur again. We apologize for the inconvenience and will continuously strive to provide a stable, secure and fast network for our clients.

Network Maintenance

Status Updates | Hostineer — Fri, 01 Jun 2018 17:35:29 +0000

Please be informed that our data center will be performing Network maintenance as described below. During the maintenance window indicated above, Network Engineers will be performing work on the core. 1am EDT (-0400 GMT) to 2am EDT will experience a full network outage and 2am EDT to 6am EDT you may experience brief periods of packet loss and latency. If you have any questions about this maintenance, please contact our 24x7 Service Desk. If all goes well we fully expect the network upgrade to be complete by 2:30am EDT.

MySQL failed ASR

Status Updates | Hostineer — Thu, 17 May 2018 17:36:00 +0000

Failed automated system recovery detected for MySQL on Atlas. Investigating further. 1:46 PM EDT- localized InnoDB pagefile corruption. Issue addressed, another MySQL restart necessary and each restart, given the size of the database server, takes about 5-8 minutes to complete. 1:54 PM EDT- MySQL immediately shutting down after starting up. Investigating further. 1:58 PM EDT- MySQL service restored. Initial cause for startup -> shutdown cycle still under investigation; appears to be linked to systemd process manager. 2:05 PM EDT- systemd service definition configuration changed using Luna, another platform as reference. MySQL comes up without issue now.

Non-specific PHP page hard locks

Status Updates | Hostineer — Tue, 24 Apr 2018 17:22:00 +0000

Over the last month, 2 reports have been received of PHP pages failing to load on Atlas. This is an isolated incident affecting 1 specific page at a time. It has been determined to be a race condition within PHP that is undergoing further investigation. The following non-essential PHP extensions have been temporarily disabled to determine the root case: - apcu

Network subnet migration

Status Updates | Hostineer — Sat, 24 Mar 2018 00:58:33 +0000

On Tuesday, March 27 beginning at 5 AM portions of the network will be migrated to a new network core. All migrations should conclude within 30 minutes. 60 second outages are anticipated for each segment of the network as it is migrated over to the new core.

Null-route on v6+ servers

Status Updates | Hostineer — Sat, 24 Feb 2018 14:22:00 +0000

Network outage reported on Atlas, Luna, and Sol platforms. Investigation with the NOC indicates a null-route was applied upstream to stem a deluge of packets originating from these servers on a memcached 0-day amplification attack. memcached access has been restricted to local-only. Moreover, for any users running memcached on your account, you are encouraged to switch to socket-based. ``` memcached -s /tmp/memcached.sock ``` Then to access memcached from a PHP application change relevant code to use the socket rather than 127.0.0.1: ``` $memcached->addServer($_SERVER['SITE_ROOT'] . '/tmp/memcached.sock'); ``` Ruby, Python, Node, Go, and shell tasks do not require usage of the environment variable, "SITE_ROOT".

Spectre/Meltdown Kernel Upgrades

Status Updates | Hostineer — Thu, 04 Jan 2018 08:41:00 +0000

Intel has recently disclosed a significant vulnerability in memory management leading to creation of two separate exploits named [Spectre and Meltdown](https://www.theregister.co.uk/2018/01/04/intel_amd_arm_cpu_vulnerability/). A kernel upgrade has been performed on all affected servers as of 3:40 AM EST (-0500 GMT). Until a better hotfix is provided, this will remain in place. Reports indicate a 17-23% [performance penalty](https://www.postgresql.org/message-id/20180102222354.qikjmf7dvnjgbkxe@alap3.anarazel.de). Newer platforms built on flexible CPU allocations (cloud-backed platforms) have been allocated an additional 25% CPU to mitigate such impact. **Update**: Jan 8 3:52 AM EST (-0400 GMT) A second kernel update has been applied that addresses [Spectre vulnerability](https://en.wikipedia.org/wiki/Spectre_(security_vulnerability)).

Network Outage

Status Updates | Hostineer — Sun, 17 Dec 2017 11:40:00 +0000

7:30 AM EST: external report received, high rate of packet loss on switch side. Investigating. 8:05 AM EST: DoS confirmed. Working with upstream provider to null-route some of the offending IP addresses as well as remove the expected target from our network. 8:48 AM EST: DoS has been contained. Network connectivity operational once again. Network connectivity will continue to be monitored over the next 24 hours. 9:07 PM EST: Received word from the data center that the origin was not an external DoS, but an internal by an improperly configured switch on a neighboring subnet: > This problem looks to have been caused by a new customer that moved in today. Their incorrectly configured switch caused a Spanning-tree reconvergence that sends out alot of traffic across the network. We have move their uplink to their Firewall/Router combo like it should have been as the permanent resolution. Sorry about the trouble, let us know if you see any other problems. **Update** 1:38 AM: Network problem has relapsed. Data center network engineers investigating. 3:19 AM: Network engineers still working on troubleshooting the origin. 3:43 AM: Network back up. Continuing to monitor.

Atlas DoS

Status Updates | Hostineer — Sun, 17 Dec 2017 00:00:00 +0000

Atlas is undergoing a distributed denial of service attacks from a variety of AWS nodes. The attacker is using a common fingerprint in these attacks, which are being patrolled to pre-emptively block to minimize further network disruption.

Atlas network stability problems

Status Updates | Hostineer — Wed, 13 Dec 2017 11:35:00 +0000

**7:35 AM**: third-party monitoring triggered a network alert for Atlas. **7:45 AM**: kernel panic in network device. New network device assigned to Atlas. **8:01 AM**: kernel panic on new network device. Bringing up new kernel. **8:10 AM**: new kernel on hypervisor. Network appears stable. Monitoring for the next 12 hours.

Core upgrade and additional 10G bandwidth

Status Updates | Hostineer — Thu, 30 Nov 2017 20:35:36 +0000

**12/4/2017 4:00 AM EST - 12/4/2017 6:00 AM EST** During the maintenance window indicated, Network Engineers will be performing work on the core network that provides your Internet uplinks. You may experience brief periods of packet loss and latency during the maintenance windows. If you have any questions about this maintenance, please contact our 24x7 Service Desk and reference CR-121417.

Panel outage

Status Updates | Hostineer — Mon, 13 Nov 2017 13:00:00 +0000

*3:00 AM EST (-0500 GMT):* Panel outage reported. *8:45 AM EST (-0500 GMT):* Panel functionality restored. Root cause: Luna serves as the default [panel reverse proxy](https://github.com/apisnetworks/cp-proxy) endpoint. Unless logged in prior to the outage, content would have been served from Luna. All servers participate in nightly updates; once code is signed off from development and pushed to production, panels update with the new code at ~3 AM EST. Sunday's change relocated Apache configuration from per-dir .htaccess to httpd.conf for improved performance. Luna diverged from the codebase thus requiring merging changes manually, which only mandates when incurred. As changes were replayed into the codebase, .htaccess had been removed first - resulting in the 404 errors - prior to manual merger effectively leaving the commit in a broken state.

Hurricane Irma Outage

Status Updates | Hostineer — Mon, 11 Sep 2017 19:27:00 +0000

4:27 PM EDT: weather from Hurricane Irma has briefly taken the data center offline. All servers automatically regained power after the momentary loss, except for Apollo and HV2, which houses Atlas, Delta, the billing server, and an internal nameserver. Both server configurations were corrected by 4:50 PM. Apollo used an old bonding setup no longer in service after the 10G network upgrade. HV2 did not expose virtual interfaces used by each VM for networking. These have been corrected in the future going forward. Irma's impact will continue to be monitored for the remainder of the day. 2017/09/12 12:06 PM EDT: data center continues to run on generator power until utility power can be stabilized. Generators include over 36 hours of fuel remaining. No further outages are anticipated once the data center switches back to utility power.

Rolling panel outages

Status Updates | Hostineer — Sun, 03 Sep 2017 23:04:00 +0000

Beginning now, 7 PM EDT (-0400 GMT) all control panels will be undergoing several internal upgrades resulting in periodic outages as upgrades are rolled out to each server. The upgrade process is expected to last 2-3 hours. 8:41 PM: upgrade window concluded.

SSL Reissuance Failure

Status Updates | Hostineer — Fri, 04 Aug 2017 19:01:00 +0000

As reported by a client, SSL certificates issued over the last 90 days were not auto-renewed on scheduled. All certificates were reissued during the last [major CP update](https://updates.hostineer.com/2017/05/launchpad-update-final-svn-release/). According to UTC timestamps, there was a ~30 minute window between expiration and resolution of the certificates. All certificates that have expired or were within 10 days of expiry have been automatically renewed at this time.

Upstream DNS authority outage

Status Updates | Hostineer — Fri, 14 Jul 2017 04:36:00 +0000

At approximately 1 AM EDT (-0400 GMT) monitoring picked up incorrect reverse DNS results upstream from our delegative authority, ns1/2/3/4.networktransit.net. The issue has been traced back to inconsistent DNS records for networktransit.net as well as incorrect PTR delegation. This has not had an immediate impact on email, because PTR records are cached for up to 24 hours. By 1 PM EDT the problem still persists waiting on a solution upstream. At this time there is limited impact to email deliverability as a mail server checks the IP address matches has a hostname and the hostname matches the IP. **3:18 PM EDT**: further digging has yielded that DNS delegation on 64.22.68.1/24 is incorrect in networktransit.net's zone definition. Waiting to hear back upstream. ``` 68.22.64.in-addr.arpa. 86400 IN NS ns1.networktransit.net. 68.22.64.in-addr.arpa. 86400 IN NS ns2.networktransit.net. 68.22.64.in-addr.arpa. 86400 IN NS ns3.networktransit.net. 68.22.64.in-addr.arpa. 86400 IN NS ns4.networktransit.net. ;; Received 131 bytes from 199.180.180.63#53(199.180.180.63) in 227 ms 1.68.22.64.in-addr.arpa.68.22.64.in-addr.arpa. 86400 IN NS ns1.apisnetworks.com. 1.68.22.64.in-addr.arpa.68.22.64.in-addr.arpa. 86400 IN NS ns2.apisnetworks.com. ;; Received 117 bytes from 205.251.138.2#53(205.251.138.2) in 129 ms ``` **7:02 PM EDT**: still working with upstream vendor waiting on response. Confirmed with data center vendor reported an "outage" last night. Whatever that entails is unknown; however, likely to be related to the zone misconfiguration that also affects other class B IP addresses in the neighborhood. **July 15, 12:22 AM EDT**: no response from upstream NOC (NetDepot) that manages DNS. Spoke with data center, NS records to mail servers were converted to PTR records around 10:30 PM. No change reported when querying ns1/2/3/4.networktransit.net. If this problem persists into tomorrow, servers may begin relaying mail through an external network, which will cause issues with SPF records. Hope for the best. ``` # dig @205.251.138.2 -x 64.22.68.2 ; <<>> DiG 9.9.4-RedHat-9.9.4-38.el7_3.3 <<>> @205.251.138.2 -x 64.22.68.2 ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41113 ;; flags: qr aa; QUERY: 1, ANSWER: 0, AUTHORITY: 2, ADDITIONAL: 0 ;; QUESTION SECTION: ;2.68.22.64.in-addr.arpa. IN PTR ;; AUTHORITY SECTION: 2.68.22.64.in-addr.arpa.68.22.64.in-addr.arpa. 86400 IN NS ns1.apisnetworks.com. 2.68.22.64.in-addr.arpa.68.22.64.in-addr.arpa. 86400 IN NS ns2.apisnetworks.com. ``` **12:56 AM EDT**: reporting PTR records on ns1.networktransit.net with a dangling NS authority record for ns2 still present (which usurps authority). Hopefully a step in the right direction and just a matter of cache. ``` # dig @ns1.networktransit.net +norec -x 64.22.68.2 ;; QUESTION SECTION: ;2.68.22.64.in-addr.arpa. IN PTR ;; ANSWER SECTION: 2.68.22.64.in-addr.arpa. 86400 IN PTR image.apisnetworks.com. ;; AUTHORITY SECTION: 2.68.22.64.in-addr.arpa.68.22.64.in-addr.arpa. 86400 IN NS ns2.apisnetworks.com. ``` **1:06 AM EDT**: rDNS confirmed in place for mail servers. Mail will continue to flow over the next 2-4 hours to its intended recipients. This fix will remain in place until the NOC corrects its PTR record delegation. Updates will be posted as they come through. **July 17, 3:09 PM EDT**: No update yet from NOC. Mail is continuing to flow through as expected; however, reverse DNS assignment is still invalid which has minimal impact. **July 19, 2:52 PM EDT**: DNS still not restored. Escalated incident with Zayo supervisor. Supervisor has informed us that through confirmation with engineering that NetDepot changed DNS backends on Friday. The transfer has not gone over as smoothly as expected. Furthermore, NetDepot will not address tickets from Zayo unless they come from Zayo's Accounts Payable department. Supervisor in process of working with engineering team to open a ticket with NetDepot from Zayo's AP department. Incidentally, Zayo now owns the data center that NetDepot once operated from and NetDepot still has a presence there. NetDepot owns the IP address space. What fun! **3:21 PM EDT**: rDNS coming back online. Issue appears to have been resolved. **July 20, 2:20 PM EDT** Periodic timeouts reported on ns4.networktransit.net. Escalated issue with NOC, awaiting response. Remaining nameservers properly forward rDNS to the correct authoritative nameservers (ns1/2.apisnetworks.com). **3:58 PM EDT** rDNS appears to be operational once again across all 4 nameservers.

Network Outage

Status Updates | Hostineer — Mon, 03 Jul 2017 13:53:00 +0000

Investigating outage as of 10:53 AM EDT, 11:35 AM, resolved. Root cause misconfiguration of a VLAN ACL

Network Upgrades, June 27 - June 29

Status Updates | Hostineer — Sat, 17 Jun 2017 18:54:37 +0000

During the period between June 27 - June 29, various servers will be scheduled for network upgrades. All upgrades will occur after 12 AM EDT (-0400 GMT) and should result in less than 30 minute outage for each component scheduled. Upgrades include improved virtualization (SR-IOV) and 10GbE for newer platforms. All remaining connections will be at least 1GbE. Affected components include: * Network uplink * Helios - 1GbE upgrade * Aleph - 1GbE upgrade * Apollo - 1 GbE upgrade * Image - 1 GbE upgrade * HV1 - 10GbE hardware upgrade * Luna * Sol * HV2 - 10GbE hardware upgrade * Atlas * Delta * Knox * Mail filtering - 10GbE hardware upgrade * Backups - 10GbE hardware upgrade More details will be posted as windows are dialed in.

Luna kernel upgrade

Status Updates | Hostineer — Fri, 16 Jun 2017 17:12:50 +0000

Luna will be taken down tonight, Saturday June 17 at 12 AM EDT (-0400 GMT) for a kernel upgrade. During this time additional evaluation will be done on OverlayFS. Expect a 2-5 minute outage during this window.

MySQL Outage

Status Updates | Hostineer — Sat, 03 Jun 2017 11:45:00 +0000

Helios experienced a MySQL outage beginning at approximately 8:45 AM EDT (-0400 GMT) that extended until 11 AM EDT, when the issue was fully resolved. MySQL has suffered from a mutex bug, since 4.0 in 2003 that can exist when a few conditions are met: * account exceeds its storage quota * multiple queries are issued to a database that is bound by the exceeded storage quota * a query is issued when another query, which cannot execute due to a blocking write, expires The lock, which should exist on the database, its tables, and nothing more leaks out to lock all tables across all accounts. It's a serious bug, which is why over the years checks have been created and reimplemented to detect and automatically restart the database service when such a condition is encountered. Presently, an internal check runs in 3 minute intervals to query the number of concurrent database connections. If that connection count exceeds reasonable operation and the system load is normal, then the mutex bug has struck and it restarts automatically. What happens when, during a 3 minute window, all available concurrent connection slots go from < 50> 50), then it is assumed the mutex bug has struck again. Instead of querying the connection count, which will fail, a restart is now issued automatically. --- Helios is the oldest server still continuously managed under Hostineer. Likewise Helios runs some pretty old software including Percona MySQL 5.5 that still suffers from the mutex bug. Newer servers run MariaDB and there is an off-chance this bug doesn't exist with that variation of MySQL, but only time will tell. Moving forward, this internal check has been corrected.

Helios unresponsive

Status Updates | Hostineer — Wed, 24 May 2017 18:09:00 +0000

2:09 PM EDT (-0400 GMT): Investigating 2:12 PM: ssh unresponsive. Taking down to single-user mode for further investigation. 2:21 PM: libcgroup PAM module found to generate segfaults after a routine package upgrade. This in turn was halting PAM from evaluating its stack on user login. Resolved and system is back up and running.

Credit card processor rejections

Status Updates | Hostineer — Thu, 11 May 2017 04:06:00 +0000

Credit card processor is currently rejecting all CC transactions with a 461 response code. In touch with the processor who is in touch with the bank. Nightly billing has been postponed until resolved tomorrow. May 19: due to unforeseen circumstances, we are abandoning our current card processor is switching systems. Billing is on target to be restored by Monday, May 22. May 21 12:29 AM EDT: billing system is functional.

Launchpad Upgrades

Status Updates | Hostineer — Sat, 06 May 2017 16:57:00 +0000

12:57 PM (-0400 GMT): Launchpad is presently undergoing [planned upgrades](https://updates.hostineer.com/2017/05/launchpad-update-final-svn-release/). No ETA on resolution. **4:58 PM EDT**: Maintenance window concluded.

Data center outage

Status Updates | Hostineer — Thu, 04 May 2017 02:10:00 +0000

Data center outage. 11:10 PM EDT (-0400 GMT) **11:35 PM EDT (-0400 GMT)**: back up. Waiting on RFO

Luna Automated System Recovery

Status Updates | Hostineer — Fri, 21 Apr 2017 21:37:00 +0000

Kernel panic on stall with 4.9. Appears to be [known problem](https://bugzilla.redhat.com/show_bug.cgi?id=1421375). Investigating. **6:08 PM EDT**: Kernel patch level is 2 releases behind. Conferring with the changelog, there are a couple changes that may explain the stall. Scheduling kernel upgrade tonight at 12 AM EDT. **12:10 AM EDT**: Kernel upgraded. Monitoring for any status changes over the next 48 hours.

Kernel upgrade

Status Updates | Hostineer — Thu, 13 Apr 2017 19:01:00 +0000

Sol, Luna, and Atlas will undergo a kernel upgrade tonight at 1 AM EDT (-0400 GMT) to address a critical flaw in the kernel ([CVE-2016-10229](https://access.redhat.com/security/cve/cve-2016-10229)). Expect a 1-2 minute outage as the servers are rebooted. **1:02 AM EDT**: All systems operational.

Panel Upgrades

Status Updates | Hostineer — Sat, 25 Mar 2017 16:00:00 +0000

Launchpad will undergo a variety of upgrades over the weekend of March 25, including a full redesign. Expect intermittent outages. Upgrades will only affect [v5+ platforms](https://kb.hostineer.com/platform/determining-platform-version/). 2:28 PM EDT (-0400 GMT): panel upgrades underway. 7:15 PM: all eligible servers upgraded (v5+). Continuing to monitor for any issues missed over the next 24 hours. Use the "Feedback" option in the nav menu to report any issues from here onward.

Apollo outage

Status Updates | Hostineer — Sat, 18 Feb 2017 18:10:00 +0000

Apollo unresponsive, investigating caused by a deadlock following cancellation of find command, a bug noted on 3.18 kernels. Further, remote access on Apollo is offline, assuming it went down with the data center power outage on 2/14. Remote hands are taking care of power cycling the server at this time. 1:30 PM - server back up, remote access firmware reflashed. We're also changing how account suspensions happen, because before today an account suspension persisted the web server config. If an account is suspended and IP address recycled for use elsewhere after 6 months, when a server comes up from boot that IP address is no longer bound to the server resulting in a critical error. An account once suspended will have its Apache configuration also removed to resolve such situations.

Data center power outage

Status Updates | Hostineer — Tue, 14 Feb 2017 22:40:00 +0000

Updates from the data center. As an aside, we were back online by 6:30 PM EST: **6:20 PM EST** > At approximately 5:40 EST, we experienced a temporary utility interruption at the data center. This temporary utility interruption caused an unknown error in our UPS which resulted in a power outage to your environment. zColo Operations are diligently working to restore power to your environment. Additional updates will be provided when available. **6:41 PM EST** > As we continue to troubleshoot the power issue affecting your data center space. Please be aware there will be some delays on ticket requests as we work through the power issues. We are working to bring the power back online safely and as quickly as possible. Our UPS vendor has been dispatched to assist us in troubleshooting the issues. Additional updates will be provided when available. **7:30 PM EST** > We are still diligently working to restore power to your environment. Our UPS vendor has an estimated time of arrival of 1 hour and 30 minutes. Please be aware there will be some delays on ticket requests as we work through the power issue. Additional updates will be provided when available. **8:33 PM EST** >Unfortunately, we are still working diligently to restore power to your environment. Our UPS vendor has an estimated time of arrival at 9:30 EST. Please be aware there will be some delays on ticket requests as we work through the power issue. Additional updates will be provided when available. **11:01 PM EST** > We have resolved the issue that may have impacted your services and believe all service levels are now restored. If you are still experiencing issues, please reach out to us immediately. An RFO will be available within 72 hours upon request. **2017/02/17** An RFO has been published from our data center. A confidentiality agreement has been tied to it, so open a ticket in the panel, as an affected third-party to request the RFO. --- On our end, misconfiguration on HV1 that required manual configuration changes to Sol and Luna. Other servers came back up as soon as power was restored. We're continuing to evaluate the overall response on our end from this disaster recovery. If you have any issues, open at ticket in [Launchpad](https://cp.hostineer.com) or drop us an email at help@hostineer.com. But please *don't call* in these situations, it takes hands off fixing major issue and instead puts them on addressing public issues to people individually! #### Addendum * An above average rate of spam flowed in for ~75 minutes. Spam filtering uses an aggregate memory-backed DB on another server, which went down with the power outage. MySQL takes some time to recover its internal InnoDB records before fully coming online, which is why DB outages take 4-5 minutes to stabilize. Further, because the system uses SysV-style startup scripts, without dependencies, rc.local fired, which restores from backup, before mysql fully started up. Adjusted to add a 2 minute sleep before attempting to restore filter data. * On v5 and earlier platforms, all dedicated IPs were not brought up on boot resulting in prolonged outages for customers on these servers. This was caused by oversight in changing an 8 year old startup routine, specifically: system("[[ ! -f /home/virtual/"$3"/info/disabled ]]"). While a successful system exit code is 0, which system(*CALL* == 0) translates to true, system(!*CALL* == 0) translates to 1 == 0 in the expected state. This issue did not affect platform v6+ and above. * CP reverse proxy failed to come up, because the path on the reboot cron was incorrect.

Backporting concurrency deadlock check

Status Updates | Hostineer — Thu, 02 Feb 2017 12:10:00 +0000

On February 2 beginning at approximately 7:10 AM EST (-0500 GMT) and concluding by 7:35 AM EST (-0500 GMT), Sol experienced connectivity issues to MySQL. The root cause is a deadlock bug that can be triggered if a few conditions are met. This bug has been a concern since it was first encountered with MySQL 4.0 released in 2003; since then, steps have been taken to reduce its exposure. Sol uses a second generation monitor that works more efficiently, tracks a wider array of defects, and can detect service flaps. Up to today, it did not detect the deadlock bug. That check has been ported to Sol and newer platforms, which will resolve incidents like this going forward. **For those curious about the bug conditions**, (1) table storage engine must be MyISAM, (2) user must be over quota, (3) identical *UPDATE* or *INSERT* queries must be issued within a millisecond of each other, (4) another *UPDATE* or *INSERT* query must be issued by another account in between those 2 queries and before the queries are postponed due to quota overages.

Luna automated system recovery & reboot startup

Status Updates | Hostineer — Sat, 28 Jan 2017 13:50:00 +0000

At 9:50 AM EST (-0500 GMT), Luna underwent an automated system recovery that necessitated a fast 20 second reboot. Upon rebooting, some services were incorrectly booted or not booted at all: **cron tasks** depend upon a state marker file in /var/run, which is typically a tmpfs mount. In synthetic filesystems it is backed by disk storage that persists across reboots. Because the marker persists, crond on boot doesn't understand this is the first boot and ignores ```@reboot``` tasks. Resolved. **mysqld** checks to see if it is running before attempting to restart, which is necessary to prevent InnoDB corruption. A known issue occurs when mysqld, on startup, performs a lengthy scan of the InnoDB journal, then a second service check fires, determines mysqld is not running, starts a second mysqld process while the initial mysqld is still recovering the InnoDB journal, then the second mysqld process binds to the known MySQL sockets (:3306, mysql.sock) without InnoDB support. A prerequisite check is made to ensure this scenario doesn't happen, as it always requires human intervention to fix, but that failed this morning. Root cause is still under investigation. **11:10 AM EST:** Resolved. lsof arguments to -F (field selection) changed between 4.87 and 4.82 causing the InnoDB lock query to fail.

Cabinet Migration

Status Updates | Hostineer — Thu, 26 Jan 2017 03:00:00 +0000

All servers will be taken offline beginning **Saturday, January 28 at 10 PM EST (-0500 GMT)** to migrate to another cabinet in the data center. Anticipate a 30-90 minute window as all servers are deracked and reracked to another location in the main annex. **12:02 AM**: still attempting to remove old rails from rack; some screws are stripped and those are being tended to now. **12:17 AM**: network online, new uplink is GigE. Continuing to cable servers, then will power on **12:41 AM**: servers coming back online **1:34 AM**: everything back online, maintenance window concluded. **2:08 AM**: another bug preventing mysql to startup on systemd-based platforms (v6.5+) has been identified and corrected.

Luna deadlock

Status Updates | Hostineer — Tue, 10 Jan 2017 14:22:00 +0000

Unresponsive. Investigating... 9:26 AM- back up, increased RAM and CPUs available to VM. 9:43 AM - bringing up new kernel to rule out any kernel issues 9:48 AM - emergency maintenance window concluded. Will continue to monitor. **Update** January 11, 10:23 AM - still seeing load surges, albeit at a less frequent rate. Further investigating... 1:45 PM - turned on process accounting to get a better snapshot if the problem continues to persist. Performance impact will be minimal.

Kernel upgrade

Status Updates | Hostineer — Sat, 07 Jan 2017 16:29:00 +0000

Atlas is being taken down now for a kernel upgrade to 4.8.16

Sol DDoS

Status Updates | Hostineer — Mon, 19 Dec 2016 20:11:00 +0000

Sol is experiencing a distributed denial of service attack. Investigating. 3:17 PM EST update: resolved.

Luna

Status Updates | Hostineer — Tue, 06 Dec 2016 23:11:00 +0000

MySQL outage, specific to 10.1.19 rolled out last week. Segfault on innodb recovery. Investigating further. 6:46 PM EST - corruption on recovering one or more InnoDB tables in the log. Moving forward with disabling some databases to tease out what IBD tables are problematic. 6:55 PM EST - corrupted database teased out. Informed account holder 7:13 PM EST - client database responsible for InnoDB crash has been wiped and restored from backup thereby removing corrupted pages from InnoDB pool. Continuing to monitor MySQL closely.

Aleph kernel panic

Status Updates | Hostineer — Mon, 05 Dec 2016 17:57:00 +0000

Kernel panic. 1:03 PM - resolved, system running. Cause under further investigation.

Sol DoS

Status Updates | Hostineer — Tue, 29 Nov 2016 04:05:00 +0000

At approximately 6:05 PM EST, Sol experienced a persistent brute-force attack on multiple web resources from a small collection of IP addresses. Typically, these are picked up and blocked without affecting performance beyond a minor blip as the system identifies and responds to the threat. A bug was uncovered during IP address matching (grep/egrep pattern distinction) that rendered automatic DoS protection ineffective. The issue has since been resolved and propagated to other servers. As of 6:20 PM EST, the system is considered stable.

Helios SSL configuration

Status Updates | Hostineer — Wed, 23 Nov 2016 02:47:00 +0000

HTTP server presently down due to SSL configuration changes. Investigating further.

Dirty Cow kernel upgrades

Status Updates | Hostineer — Sun, 23 Oct 2016 21:19:00 +0000

Given the severity of the Dirty Cow vulnerability and variants that are and are not effective against different variants of RedHat Enterprise Linux, which we use, all servers are undergoing a proactive kernel upgrade this afternoon. Expect a < 5 minute outage for newer generation servers (v6+) and a 10-15 minute outage for older generation (pre v6) servers. Luna has already been upgraded, so no further action will be taken on Luna. 3:38 PM EDT - maintenance window concluded

Apollo outage

Status Updates | Hostineer — Sun, 23 Oct 2016 20:56:00 +0000

Apollo outage, investigating... 1:00 PM EDT - Apollo back up, stemmed from live testing of a variant of the Dirty Cow exploit. Not successfully leveraged, but does induce a kernel panic. Upgrading kernels on all servers...

Luna kernel upgrade

Status Updates | Hostineer — Sat, 22 Oct 2016 08:35:00 +0000

Luna is being taken down immediately for a kernel upgrade in response to the Dirty Cow exploit. Luna is the only RHEL7 server and the only one vulnerable to the exploit.

Aleph network speed

Status Updates | Hostineer — Wed, 12 Oct 2016 16:28:00 +0000

Poor network throughput has been noticed on Aleph. A kernel upgrade with a later firmware is being tested to verify whether this resolves throughput. A brief (~5 minute) reboot has been scheduled immediately. Update: network throughput has returned to typical levels. Will continue to monitor.

Augend retirement

Status Updates | Hostineer — Tue, 11 Oct 2016 20:00:00 +0000

Beginning today, October 11, Augend will transition towards retirement. All accounts on Augend will be migrated off to neighboring servers to make room for a new hosting platform. A final, permanent migration will occur this Sunday, October 16. Thank you for 9 fantastic years of service, Augend! 2016/10/19 10:17 AM EDT: all accounts migrated off of Augend. Server powered down.

Luna kernel upgrade

Status Updates | Hostineer — Sat, 10 Sep 2016 11:10:00 +0000

Luna will be undergoing a kernel upgrade within the next 3-5 minutes to further troubleshoot a peculiar SSL negotiation bug affecting 1 account. 3:18 AM EDT - upgrade complete, server stable.

Sol performance disturbance

Status Updates | Hostineer — Wed, 17 Aug 2016 19:07:00 +0000

Sol is experiencing temporary performance degradation that is under further investigation. 12:15 PM EDT - adjusted available CPUs to Sol to alleviate CPU bottleneck. Performance should return to normal.

Aleph outage

Status Updates | Hostineer — Sat, 30 Jul 2016 07:32:00 +0000

Aleph is currently non-responsive. Investigating. 11:56 PM - Stabilized. Investigating... Kernel panic on IPMI. 1:50 AM - IPMI appears to be corrupted. Server is fully operational and will continue to investigate further.

Billing system changes

Status Updates | Hostineer — Sat, 23 Jul 2016 21:51:00 +0000

Improvements are being made to the billing backend. During this window, expected to conclude by 6 PM EDT (-0400 GMT) today, subscriptions may not be altered within the control panel or billing portal. 5:58 PM - billing changes finished and maintenance window concluded

Augend kernel upgrade

Status Updates | Hostineer — Wed, 20 Jul 2016 08:27:00 +0000

Augend is being taken down briefly to bring up a new kernel. Estimated outage is 3-5 minutes.

Recurrent high load averages

Status Updates | Hostineer — Sun, 03 Jul 2016 23:58:00 +0000

We are continuing to see recurrent high loads on Sol periodically. This is under further investigation. 6:05 PM EDT - scheduled a kernel reboot later tonight at 12 AM EDT, anticipate a 2-3 minute outage while the new kernel is brought up. 11:58 PM EDT - taking Sol down for upgrade 12:10 AM EDT - taking longer than expected; Sol coming up for final reboot... 12:11 AM EDT - Sol up. Monitoring for improvements over the next 48 hours. 2016/07/06 11:29 AM EDT - ending monitoring of Sol; server appears stabilized.

HTTP outage

Status Updates | Hostineer — Sat, 25 Jun 2016 02:58:00 +0000

Identified and resolving an issue with SSL configuration on Helios 6:10 PM- issue has been resolved.

Backend anomalies

Status Updates | Hostineer — Sat, 25 Jun 2016 02:20:00 +0000

Engineers have identified irregular behavior within the control panel for certain backend operations (DB creation, SSL certs) over the past 24 hours. Any operations that may have failed within the 24 hour window should be done again now that the issue has been resolved.

Spam filtering degraded

Status Updates | Hostineer — Mon, 13 Jun 2016 17:07:00 +0000

A higher rate of false negative emails has been reported over the last 3 hours. Engineers have investigated the issue. Filtering has been restored and will gradually improve over the next 72 hours.