Tucson, Data Centers and the AI “Bubble”

I’m the Vice President of Operations for a Data Center Provider in Tucson – It’s easy to find out which one, I just keep them purposefully separate – and we’ve had a lot talk lately within the community about Data Centers and the resource consumption they demand, specifically in Tucson around Project Blue.

One of the big demands of some facilities is water, if they’re going to be doing GPU workloads or just intense workloads and turn to child water ( they should… it’s very efficient at scale ) the initial proposal was for an open loop water cooled chiller plant. Because of the low humidity in Arizona, and the initial low cost to implement such a system it would make sense to go open loop but there’s one issue…. water consumption. Now, given that we are in the desert, and if you had done any research in the city, you’d know that you are not going to have a warm reception to high water usage. They did not do that and so the community rejected the offer.

Closed loop cooling is more expensive up front and not quite as efficient but they do last longer and only require small amounts of makeup water after fill.

Finally you can do chilled water or glycol with a refrigerant based system, again just sticking with fluid type systems and not direct expansion systems.

We use direct expansion right now as our critical load is only 250kW at the moment, and that’s not enough to build out more than 1 chiller from a cost perspective. We will install a glycol loop for GPU workloads and direct to chip cooling before convert our DX units.

All this to say, being careful of your surroundings and environment makes a big difference in what type of headwinds you face in a given project. Perhaps if the team behind it had more than 6 months of experience working together and spent some time in the community before the initial design they could have known this. Furthermore the power load, some 650MW is more than the city currently consumes. A consumption that is already hare for the local utility to provide reliable power to, we have hundreds of power monitors across southern AZ and it’s not great. Additionally we ourselves have to routinely go to generator in unplanned events every year due to utility failures, in the city. I don’t think that it’s rational to think that the local utility was going to be able to double their power supply in 2 years without community impact.

The AI boom is not a bubble – I don’t think, but it will be interesting to see how much capital comes calling back and how soon, there’s a lot hanging out there right now and I’m not sure what they are going to reasonably do for power. Microsoft bought a nuke… others are using gas turbines on site through various loopholes. It’s interesting… We have been looking at the same requirements and we are going to install about 300kW of solar in the city to offset some load. We have about 250kW of critical power shell available right now but hopefully in the near future we will have about 10MW available but, hopefully, in a sustainable manner.

Some thoughts on the world today.

I don’t like to make political posts of any kind on this page, it is sincerely geared towards just notes and tips and tricks for technical topics, it is not meant to be anything beyond that however, I think it’s important for the technical community to safeguard certain data that is attempting to be removed from access by citizens of the US.

If you are of the ability and have the capability and you see something that is being scrubbed, take that data, and share it with the world elsewhere, things that are resources for people and the Internet as a whole should not be censored, ever. Access to the Internet should not be controlled by a governmental body, ever.

Data collection is bad, but removing the choice of what you consume in order to control the narrative is even worse, do not accept actions of your government to control the information for which you have access, resist.

That is it, thanks.

Extending an LVM / XFS mount/partition online in Rocky Linux

Scenario: You have a VM and you’ve increased the disk size in the hypervisor and want to extend the LVM and partition in the guest.

LVM is the last partition so we can grow it versus creating a new partition and adding it to the VolGroup:

(parted) p                                                                
Model: Xen Virtual Block Device (xvd)
Disk /dev/xvda: 335544320s
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start     End         Size        Type     File system  Flags
 1      2048s     2099199s    2097152s    primary  xfs          boot
 2      2099200s  167772159s  165672960s  primary               lvm

The easiest way to extend this is via the growpart tool, found in the cloud-utils-growpart package in AppStream

dnf install cloud-utils-growpart
Last metadata expiration check: 3:13:02 ago on Fri 25 Oct 2024 04:47:25 AM MST.
Dependencies resolved.
==============================================================================
 Package                                                                               Architecture                                                            Version                                                                      Repository                                                                  Size
==============================================================================
Installing:
 cloud-utils-growpart                                                                  noarch                                                                  0.33-0.el8                                                                   appstream                                                                   35 k

Transaction Summary
==============================================================================
Install  1 Package

Total download size: 35 k
Installed size: 75 k
Is this ok [y/N]: y
Downloading Packages:
cloud-utils-growpart-0.33-0.el8.noarch.rpm                                                                                                                                                                                                                                                    78 kB/s |  35 kB     00:00    
------------------------------------------------------------------------------
Total                                                                                                                                                                                                                                                                                         47 kB/s |  35 kB     00:00     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                                                                                                                     1/1 
  Installing       : cloud-utils-growpart-0.33-0.el8.noarch                                                                                                                                                                                                                                                              1/1 
  Running scriptlet: cloud-utils-growpart-0.33-0.el8.noarch                                                                                                                                                                                                                                                              1/1 
  Verifying        : cloud-utils-growpart-0.33-0.el8.noarch                                                                                                                                                                                                                                                              1/1 

Installed:
  cloud-utils-growpart-0.33-0.el8.noarch                                                                                                                                                                                                                                                                                     

Complete!

Now extend partition #2 with the tool:

growpart /dev/xvda 2
CHANGED: partition=2 start=2099200 old: size=165672960 end=167772159 new: size=333445087 end=335544286

That’s it, the partition is now extended, next up, extend the LV:

lvextend -l +100%FREE /dev/mapper/rl-root 
  Size of logical volume rl/root changed from <71.12 GiB (18206 extents) to <151.12 GiB (38686 extents).
  Logical volume rl/root successfully resized.

Finally grow the XFS file system via xfs_growfs

xfs_growfs /
meta-data=/dev/mapper/rl-root    isize=512    agcount=4, agsize=4660736 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1    bigtime=0 inobtcount=0
data     =                       bsize=4096   blocks=18642944, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=9103, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
data blocks changed from 18642944 to 39614464

TrueNAS Core Nextcloud 23.0.2 Plugin update -> 27.0.2

A nextcloud plugin in a TrueNAS Core system that hadn’t been updated is well, a problem. If you update the Plugin it will go straight to Nextcloud 27 which well you can’t do an update like that. So here’s what we did.

Update to the latest version of 23 in the web browser and proceed with this until you get to version 25. You will notice that you cannot get past here if you’re on a Jail running 12.2 because it has PHP74 installed by default with the original Nextcloud package.

After this point you have to install PHP 8. But doing so will try and remove nextcloud-php74 package which would remove Nextcloud of course and kinda make the whole thing pointless so, we need to force that to remain installed while we gut PHP and reinstall PHP 8.

Lock the nextcloud package, we don’t want to remove it.

pkg lock nextcloud-php74

Install PHP80

pkg install php80-bcmath php80-bz2 php80-ctype php80-curl php80-dom php80-exif php80-fileinfo php80-filter php80-gd php80-gmp php80-iconv php80-intl php80-ldap php80-mbstring php80-opcache php80-pcntl php80-pcntl php80-pdo_mysql php80-pecl-APCu php80-pecl-imagick php80-pecl-redis php80-posix php80-session php80-simplexml php80-xmlreader php80-xmlwriter php80-xsl php80-zip php80-zlib

Restart FPM to reload the web UI:
/usr/local/etc/rc.d/php-fpm restart

Perform the update through to v27 from the Web UI.

Update the plugin in TrueNAS Core once the web UI update completes. THIS WILL FAIL.

Once it fails, log back into the console and manually uninstall/install the following:

pkg unlock nextcloud-php74
pkg remove nextcloud-php74-23.0.2
pkg install nextcloud-php80 nginx mysql80-server redis php80-pecl-redis wget rsync

Now, the included nextcloud version is 27.0.0 and we’re already above that and downgrading is not supported so, we need to pull the archive from nextcloud and copy over our install;

mkdir /root/scratch
cd /root/scratch
wget https://download.nextcloud.com/server/releases/latest.tar.bz2
tar -xjvf latest.tar.bz2
rsync -avz /root/scratch/nextcloud/ /usr/local/www/nextcloud/

Now we need to fix some permission:

cd /usr/loca/www
chown www:www nextcloud
cd nextcloud
chown -R www:www .htaccess .user.ini 3rdparty/ apps AUTHORS config/ console.php COPYING core/ cron.php dist/ index.html index.php lib/ occ ocm-provider/ ocs* public.php remote.php resources/ robots.txt status.php themes/ updater/ version.php

Now start redis and mysql and, restart fpm;

/usr/local/etc/rc.d/redis start
/usr/local/etc/rc.d/mysql-server start
/usr/local/etc/rc.d/php-fpm restart

All done, easy right? LOL.