Coresrv01 is our main core services server. It is a CentOS 7x server responsible for:

If there is a problem with NFS/autofs, stop the web server and then try restarting autofs.

[root@coresrv01 ~]# systemctl stop httpd
[root@coresrv01 ~]# systemctl restart autofs

# If restarting autofs hangs try

[root@coresrv01 ~]# killall -9 automount
[root@coresrv01 ~]# systemctl start autofs

# Make sure /home is mounted!

[root@coresrv01 ~]# mount /home # if that fails, try: 
[root@coresrv01 ~]# umount /home; mount zfs3:/home /home

# Then restart httpd

[root@coresrv01 ~]# systemctl start httpd

After a cold boot start portal_config_gen

[root@coresrv01 ~]# systemctl start portal_config_gen

# Check status/restart
systemctl status httpd
systemctl restart httpd

There is a nightly backup of all mysql data on mnemosyne.its.virginia.edu in /u-p_mirror/mysql/

Restore doing similar:

destination-server# ssh -l root -i rootkey mnemosyne.its.virginia.edu "cat /u-p_mirror/mysql/coresrv01_2019-04-16_03:41:39.sql" | mysql -u root -p

coresrv05 is set up and ready to go as a replacement dns/dhcp server. We just need to add the address 128.143.67.11 as an IP on coresrv05

Do the following:

ip addr add 128.143.67.11/16 dev eno1