*RESOLVED* pfs file system slow/down, 2019-04-04

2019-04-04:

We are experiencing severe slowdown on the /pfs/nobackup file system, affecting all accesses including running jobs.

This is caused by components in the storage system restarting for unknown reasons, investigation is ongoing.

*UPDATE* In order to identify what is going on we are forced to shut down the file system occasionally. The vendor is assisting in identifying and fixing the issue.

*UPDATE 20190405 00:40* The file system servers are no longer crashing/restarting and things are starting to look stable again. We will keep the batch queues stopped until the morning to make sure it really is stable.

*UPDATE 20190405 01:10* The servers are still crashing, although not as frequently. We're going to have to investigate more in the morning.

*UPDATE 20190406 10:30* Kebnekaise is now up and running again.

*UPDATE 20190406 11:10* Abisko is now up and running again.

Some files may have gotten corrupted or been lost. Please let us know if you find any problems.