*
News: SMF - Just Installed!


Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length

/opt filling on FSM appliances 1471

Super Newbie Posts: 3 Karma: +0/-0 *
/opt filling on FSM appliances
« on: November 11, 2015, 10:42:20 PM »
Hi all,

I recently performed an upgrade of our FSM appliance from 1.3.3 to 1.3.4 and during the upgrade, it looked like there was not enough space to complete the upgrade and we received a message as follows :

ATKUPD764I Update "fsm_appliance_update_preparation_1.3.4" was installed on system "Red Hat 795501M XXXXXXX 41020B52-D337-4388-9148-XXXXXXX" successfully.
ATKUPD783E An error occurred while updating "com.ibm.director.storage.storagecontrol.member.Linux_4.2.8.0-build-00057" on system "Red Hat 795501M XXXXXXX 41020B52-D337-4388-9148-XXXXXXX". Restart the Common Agent on the managed system, verify connectivity to the system, and try again. Error:
ATKUPD268E The install requires an estimated 4805 MB of free space, and the disk volume containing the directory "/opt/ibm/director/lwi" has 4200 MB of free space remaining.
ATKUPD705E The update installation request was not successful for system "Red Hat 795501M XXXXXXX 41020B52-D337-4388-9148-XXXXXXX". Search above for previous related errors, fix each error, and then retry the operation.
ATKUPD739I Collecting inventory on system "Red Hat 795501M XXXXXXX 41020B52-D337-4388-9148-XXXXXXX".

To cut a long story short, call was placed with IBM because after rebooting it, the DB2 instance did not start as per http://www-01.ibm.com/support/docview.wss?uid=nas73027b8075eeb67d986257e680042a6a9 and also managed to free up the required 605MB for the upgrade to complete.  However the call has been kept open as /opt now only has 2 GB of free space available, so when subsequent updates are to be applied in future there will be insufficient space.  After much back and forth with IBM, they have came to the conclusion that the only way to resolve the issue will be to do a full rebuild of the appliance and then reconfigure it from scratch, as the space is not cleaned up correctly when applying FSM patches and our FSM is quite old and had many patch bundles applied to it  (Originally 1.1.1 FP2 according to some notes I have on it).

I was wondering if anyone else has come across this and were they basically put in a position where rebuilding the appliance is the only way to fix it for future upgrade attempts or if they have any suggestions on what could be cleaned out of /opt without needing to do a full rebuild of the appliance ?

Cheers,

Jason
Administrator Sr. Member Posts: 309 Karma: +6/-0 *****
Re: /opt filling on FSM appliances
« Reply #1 on: November 12, 2015, 02:24:21 PM »
Hi Jason,

Before you performed the upgrade did you run...

smcli cleanupd -mva

That command is not included in the documentation but is definetely a very important step that should be included. This cleans out the entire update repository. Now, I'm not sure if the updates live under /opt but I also upgraded from 1.3.3 to 1.3.4 recently at a customer and did not see this issue so I have to believe they do.

If you have never ran this command then note that it could take an extraordinary amount of time to complete so please be patient.

After you clean out the updates, reboot the FSM, and try the update again.

However, if the FSM software cannot even start then I'm not sure this command would be available thus forcing you to rebuild the FSM.  :-\ If that is the case I am not sure what level of code your Recovery Partition is running but I would advise requesting IBM send you the Recovery media for 1.3.4  so you won't need to do a series of upgrades to your FSM post rebuild. The caveat to that is if you have a recent backup of the FSM on 1.3.3 code, if this is the case then I would request the Recovery Media for 1.3.3 just to be sure your restore will work.

Does that make sense?

Thanks!
Super Newbie Posts: 3 Karma: +0/-0 *
Re: /opt filling on FSM appliances
« Reply #2 on: November 12, 2015, 11:12:51 PM »
Hi Jason,

Before you performed the upgrade did you run...

smcli cleanupd -mva

That command is not included in the documentation but is definetely a very important step that should be included. This cleans out the entire update repository. Now, I'm not sure if the updates live under /opt but I also upgraded from 1.3.3 to 1.3.4 recently at a customer and did not see this issue so I have to believe they do.


Hi,

I had ran that command and it does actually detail it in the newer release notes that if you are concerned with space on the appliances to run that.  However on my FSM appliance, the updates now live in /updateslib.  I've attached the "du -k" output to clarify :

FSM-XXXXXXXXX:~ # df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/vdb2             10079784   5015784   4551968  53% /
tmpfs                 14137024       208  14136816   1% /dev
tmpfs                 14137024         4  14137020   1% /dev/shm
/dev/vda1              1026776     40844    933772   5% /boot
/dev/vdb3             30241636  25749192   2956248  90% /opt
/dev/vdb5              6048248   2789420   2951592  49% /var
/dev/vdb6              5043920    650712   4136988  14% /tmp
/dev/vdb7             23971336  16837968   5915676  75% /home
/dev/vdd1             82563600   5178192  73191424   7% /mirror0
/dev/vde1            123849912    988236 116570468   1% /dump
/dev/vdf1             30959676    232028  29154988   1% /updateslib

So as you can see /opt on my FSM appliance is at 90% used, and this is where my real concern is.  The FSM as it is now is working, but I am concerned with future updates, as the last update I installed wanted at least 4.8 GB free in order to proceed (and currently there is around 2.9 GB free)..  I am open to any ideas to free up space...  I've approached IBM about removing contents in /opt/ibm/director/proddata/updatedata/backup as this directory seems to contain a lot of backup versions of applications that are updated with each release, but don't seem to be removed, however IBM had no documented procedure for clearing this directory.


If that is the case I am not sure what level of code your Recovery Partition is running but I would advise requesting IBM send you the Recovery media for 1.3.4  so you won't need to do a series of upgrades to your FSM post rebuild. The caveat to that is if you have a recent backup of the FSM on 1.3.3 code, if this is the case then I would request the Recovery Media for 1.3.3 just to be sure your restore will work.


IBM are currently suggesting that we do a full reinstall of the FSM appliance from scratch and then reconfigure it from scratch as the only way to "reclaim" space in /opt.  I was just wanting to see if anyone else had come up against this issue and had workarounds to get space back in /opt for re-use without having to do a full rebuild of the FSM from scratch.  There is a bit of configuration in there currently that we don't want to have to redo again.  IBM also stated that even a Backup of the appliance and then doing a recovery of the appliance to 1.3.4 and then restoring from the backup will still consume the same amount of space in /opt as the backup does indeed backup all of the /opt directory, so have ruled that out as a way to reclaim space.

Thanks for your help though.  The suggestions did make perfect sense, although have been tried already :)

Cheers,

Jason