system news

We are currently experiencing severe problems with the pfs (parallel file system).

Both Kebnekaise and Abisko are affected, including access to the PFS filesystem from the login nodes. Either it takes a very long time to access the files or the files are not available. We recommend that you try to avoid using the PFS filesystem. 

It is working intensively to solve the problems. At the moment we have no ETA when the problems will be resolved.

Note: The batch queues for Kebnekaise and Abisko are stopped until further notice.

2019-06-13, 15:40

We are currently experiencing a slowdown of the pfs (parallel file system). This affects all access, both Kebnekaise and Abisko.

The queues have been stopped until further notice.

We are investigating and will update this news as soon as we have any information.

2019-05-10, 17:30

Rack two of Abisko went down due to an electric error within the rack and all jobs running on it was lost. We are looking into the electric problem.

This news will be updated when there is more information.

2019-04-04:

We are experiencing severe slowdown on the /pfs/nobackup file system, affecting all accesses including running jobs.

This is caused by components in the storage system restarting for unknown reasons, investigation is ongoing.

*UPDATE* In order to identify what is going on we are forced to shut down the file system occasionally. The vendor is assisting in identifying and fixing the issue.

2019-03-29:

Normal power routing restored to all nodes.

2019-03-28:

Repairs completed, switchboard powered up.

2019-03-25:

Due to component delivery delays the final steps of repair are postponed. The new date for completion is Wednesday 2019-03-27.

2019-03-20:

Replacement parts and cables are en route. We currently estimate installation and recertification of the switchboard to be finished by the end of Monday 2019-03-25.

2019-03-15:

There was a (25-minute) power outage at Umeå University campus just before 09:30. This brought down our clusters (Kebnekaise and Abisko and killed all running jobs. It also has affected the Kebnekaise login nodes.

Power is back now, and we are in the process of taking up the clusters. We will add more information when we know what happened and/or when the clusters are back up.

*Update* The reason for the power outage was a severed cable that triggered a circuit breaker in one of the universities internal power stations.

At around 19:20 today (30/1), the power to the clusters was lost. That means both kebnekaise and abisko is not running any jobs. All running jobs has has stopped. We are working on getting back the power and the cluster up. This message will be updates has we know more.

Login to the kebnekaise access nodes will stall until the kebnekaise cluster is back up. Abisko login node should work.

*Update* 2019-01-30 22:40: Abisko should be up and running again. Kebnekaise has some problems with /pfs and is still down. Probably will not be fixed until tomorrow.

We have an electrical maintenance planned in our cluster room 20190128 07:30 - 18:00

The work requires the power to be turned off and thus all clusters will need to be drained and taken down.

We have therefor set a reservation on the clusters preventing any jobs from running during that time frame.