As some of you may know we have been working on server upgrades to help improve performance and stability. This morning we moved to step 2 of 3 and implemented a Network File Server (NFS). The NFS will be used as a shared file server between each of the load balanced web servers (coming soon) so that files between servers can be kept in sync.
Everything went fine right up to the final step. After we copied all files to the NFS and did the file step to put it in place site immediately stopped showing up. It took a little while to figure out the problem but eventually we narrowed it down to how the Zend Framework (which we use as the base framework for the CMS) handles caching. It doesn't work well in an NFS environment. Once we figured that out we disabled caching and sites immediately started coming up.
It turns out that was just one of 2 problems.
We use the Zend Framework for searching too and it turns out that it also does not work well in an NFS environment. To prevent a lot of errors we've temporarily disabled searching while we sort this out. All search results return 0 matches. We are currently looking into solutions to get search back up and running. There are two general solutions... store the search index for each site in a different spot (that is part of the issue) or try an implement a different search solution. We will update here once we have figured out the best solution.
We do apologize for the interruption in service and the issue with the search. We are doing everything that we can to find a solution as quickly as possible.
Note - this only affects websites on the Web2 server.
Update 1/13/2013 7:08 pm
We got caching working again. We're storing the cache files in a different spot and it's now working great. Sites are faster again. Now, onto search.
Update 1/14/2013 8:09 am
We've noticed some latency when connecting via FTP. It would be more efficient to connect directly to the NFS. We've setup a domain that you can use to easily connect to the correct address.
You can now use ftp.mysiteftp.com as the server/domain part of your FTP connection.
Update 1/14/2013 8:22 am
Search is working again. Each site will need to be manually reindexed by going to My Apps -> Search -> Dashboard. Once there click the "Manually update the index link" and then follow directions. We are going through each affected domain and updating the index, but if you get there before we do then go ahead and update the index.
Update 1/14/2013 9:22 am
We've manually updated the search index for all sites on Web2 server that have search enabled. At this point everything should be working correctly.