Pug seems to be experiencing problems. Pug becomes unreachable for minutes at a time and then is reachable again for a few minutes. We are troubleshooting this issue now and will update you when we have more information.
Update (23:45 PST): This was a monitoring false positive. Everything is operating normally and well.
We are responding to a notification from a user and our monitoring software of an attack on our servers. We detect no crash or compromise on the server but we take any threat seriously and will update you as soon as possible.
Update (22:14 PST): A wanna-be ‘hacker’ has executed a perl script on chow which has exploited a hole in a user’s folder settings (specifically chmod 777). This caused the attacker to be able to copy their own index file over existing index.html or index.php files, though the page is a nuisance it does not pose a threat and no other data was modified or removed. We are responding to this threat and will update you when we are able to.
Update (Wednesday, March 19, 0:45 PST): We have fully locked down and assessed the damage done by this script. It is very minimal and we will be able to restore effected account’s index files from backups. we have changed all of our passwords and have run security audit tools to make sure no other holes exist and have addressed any errors. No other servers have been effected by this attack. We will update you when we begin to restore data.
Update (2:55 PST): Our backup server is up-to-date but needs to be reconfigured for the restore.
Update (4:30 PST): We have eliminated the threat and have tracked down the hole and have rectified the issue. We have also determined that our server is functioning properly and that no data was lost or stolen. The attack did not effect other parts of the server but merely replaced index files in user’s public_html folders via a perl script. With a shared hosting environment comes the need to run and provide many different services, features and freedom of control; unfortunately, with this come security risks.
Update (7:32 PST): We are beginging the restore now. In addition we are rebuilding Apache & PHP for security purposes just to be sure all bases are covered. Apache & PHP seem to be functioning fine but we would like to re-harden our server and not be sloppy.
Update (12:34 PST): As it turns out the recent rebuild of apache had a configuration issue that is now causing all sites to read Error 404. We are working to resolve this via our backups. We are restoring our backups of the home directory and once completed (user by user) the error will go away (user by user). Please be patient as there is a lot of data.
Update (20:04 PST): We have rehardened apache & PHP, during this process we were able todetermine the cause of the 404 Error, users privileges and home directory mappings are a bit screwed up. As each user accoutn is restored via backups this error will go away.
Update (Thursday, March 20, 4:56 PST): Restoring data is still underway. When a user’s data is copied over we are then fixing the permissions issue and domain names are now working one by one. We greatly appreciate your patience and are working to make sure, should this happen int he future, we are better prepared to respond quickly and with the least amount of downtime. The issue was out of our control but had our restoration procedure been more refined (a lot of user data makes this difficult) we would have had a faster restoration time.
Update (7:25 PST): Restoring user’s domains is almost complete and all domain names check out. If you are still getting the 404 error please be patient. IF you are having other difficulties, please contact us.
Update (12:30 PST): All user accounts have been restored, if your site is still down please let us know. Chow seems to be back to 100% now. We of course will be monitoring chow closely and will be performing maintenance during th next few days. We are going through every acount now to make sure everything is nominal.
On Thursday, March 13, 2008 from 8:00 PST - 17:00 PST (March 13, 2008 from 16:00 GMT - March 14 1:00 GMT) our SFO1 data center facility will be running on generator power for up to 9 hours while the power utility performs maintenance to the power supply lines on our street. There should be no impact to the power in the facility and no down time is expected. This notice is for advisory purposes only.
NOTE: This is a reschedule of the maintenance originally planned for February 22. The power utility PG&E will be performing maintenance on electrical service lines in the streets adjacent to SFO1 data center facility. As with any maintenance there are potential risks, but this is an advisory notice only as we do not expect any issues. If there are any additional questions before, during, or following the maintenance, please direct them to the Member’s Center.
Update (9:42 PST): Power maintenance is presently under way, and the data center facility is running on generator power.
The SFO1 facility will be running on generator power for up to 9 hours while Pacific Gas & Electric performs maintenance to the power supply lines on the data center’s street.
Update (16:10 PST): Transfer back to the power utility is complete and SFO1 is currently running under utility (PG&E) power. No incidents of losses of power within the data center were reported while equipment was running on diesel generators or UPS systems.
Pug will be rebooted tonight between 23:00 PST and 23:30 PST to trouble shoot an issue we are having with ClamAV. This is a planned outage and should not leave your site down for more than a minute or two. We have chosen an off-peak hour to lessen the impact of this reboot and will update you when we are rebooting.
Update (23:25 PST): We are rebooting Pug now…
Update (23:26 PST): Pug is back up and came up quick. Service will resume as normal now. Thanks!
This Friday, February 22, 2008 from 8:30 AM PST - 11:59 PM PST (Friday, February 22 4:30 PM GMT - Saturday, February 23 8:00 AM GMT)
The power utility PG&E will be performing maintenance on electrical service lines in the streets adjacent to SFO1 data center facility in order to accommodate the power needs of a large new building adjacent to our facility. As with any maintenance there are potential risks, but this is an advisory notice only as we do not expect any issues.
If there are any additional questions before, during, or following the maintenance, please direct them to our support team via the Member’s Center.
Update (Friday, February 22, 9:06 PST): Power has been transfered to backup power.
Update (12:24 PST): Power has been failed-back to utility power. However the maintenance will have to be rescheduled. More to come soon.
Update (14:45 PST): SFO1 datacenter facility transferred to generator power as planned at 9:03 AM PST.
Although the transfer was successful, during a testing phase prior to the utility power being cut, we experienced an alarm on our primary generator.
The generator fault was identified as a malfunction with the roof cooling tower and has been remedied.
Unfortunately the power utility PG&E still needs to perform maintenance to the electrical lines in the street, and so the maintenance will be need to be rescheduled.
We will provide an update once the maintenance is rescheduled.
Pug had some problems keeping its SMTP server running this morning due to a large amount of SPAM sent out by a user. We immediately suspended the user once our spam tracking software picked up the threat and then went on to take care of the SMTP deamon. Unfortunately a reboot did not fix the issue. We are looking into why exim will not stay running now…
Update (9:20 PST): It seems that there is still spam in the queue that needs to be purged causing exim resource usage to spike and die. We are doing this now.
Update (9:40 PST): All SPAM messages have been purged and exim is up and running stably again. We will monitor exim closely for the next few hours to make sure this remains the same and will update you if any issues arise again. We thank you for your patience and welcome you to contact us if you need any assistance. We at Cyberdog take SPAM related issues very seriously which is why we reacted to this issue as soon as possible. Spammers beware, your spam will not be tolerated so don’t waste your time.
Robert Davis
Network Operations Manager
We are updating the firewall on our servers to configfirewall. Due to one of our techs overlooking a requirement the firewall has caused our servers to become unreachable from the outside. We are working on this issue now.
Update: (8:10 PST): Services are back up. Try true commencing…
Update (8:54 PST): We have installed the required modules and have rebooted our server services. Although this seems to have locked us out once again. We are now going to reboot our servers completely.
Update (8:59 PST):And we’re back. The reboots were a success and the modules needed are now installed. We are not configuring our new firewalls.
Robert
Network Operations Manager
It has come to our attention that we are missing RDNS entries for our servers. Because of this e-mails are being rejected by certain service providers such as Comcast who require a valid RDNS be listed for each of our mail servers, and servers in general. We are working to resolve this issue as soon as possible.
Update (22:32 PST): Our Monitoring software and our data-center has both just notified us that server Chow is being listed on SpamCop as a potential spam threat due to our lack of RDNS entries: http://www.spamcop.net/w3m?action=checkblock&ip=64.151.102.171. However this listing is based off of the lack of RDNS entries solely and is set to expire within 12 hours if no report of spam is found to back and validate this threat. we have asked the data-center to create RDNS entries and should have them soon.
Update (Thursday, January 31, 2008, 4:32 PST): The data-center has created our RDNS entries, however they are malformed and may still cause e-mail to be rejected form some mail servers. Instead of chow having the RDNS of its FQDN the FDNS is tagged onto the front of our first 3 IP Address octets listed backwards with int-adr.arpa as the top level domain. We are working with the data-center to resolve this issue.
Update (11:15 PST): The listing on SpamCop has expired and has been removed: http://www.spamcop.net/w3m?action=checkblock&ip=64.151.102.171.
Update (2:47 PST): The data-center has fixed the RDNS issue and the changes are reflecting correctly. The issue was no top level root period (.) was given at the end of the entry, doh! Because of this the domain was thought to be a sub domain and so the traditional apra ending was appended. You always add the period! Oh well, the issue has been resolved and it was a minor mistake. The RDNS entries can take up to 48 hours to propagate world wide at which time e-mail should resume normally with all ISPs. Of course if this is not the case or if you require support please feel free to contact us via the Member’s Center.
Robert
Network Operations Manager
Chow’s FTP has been flaky and has been causing connection problems for multiple users and FTP programs. We are looking into the cause of this issue and will update you shortly.
Update (16:30 PST): This may be an issue with PASV connections to FTP. SmartFTP merely reconnects using no PASV transfer while other programs simply give up. We have updated FTP, restarted it, synced passwords etc. but are now focusing on what is causing this issue. FTP is up, and you should be bale to connect, however when getting a directory listing through PASV transfer you will experience issues. We’re still looking into this issue.’
Update (21:33 PST): We have hired a private technician on an hourly basis to fix this issue. He was able to fix it promptly and make up for the time lost by SM not being able to track down the issue. While this cost us a lot of money we received a good amount of complaints about the issue. As it turns out PASV transfer wasn’t work half the time because the firewall as blocking PASV ports, but not all of them. Firewalls can sometimes be too secure but this case was strange. Luckily the firewall has been changed and will not cause this issue in the future. If you need any help please let us know.
Chow is currently receiving an invalid license error. We are in contact with cPanel and hope to resolve this shortly. Service to sites on chow and e-mail functionality has not been lost, you will just not be able to login to your cPanel until the error is resolved.
Update (20:32 PST): The error was resolved within a few minutes of its discovery. When we changed the IP Configuration on the server the cPanel license got corrupted. A simple cPanel license update command resolved the issue.