(Hoping someone has a similar situation...)

I have a homelab with an APC BackUPS 1500 XM, a Synology NAS running DSM
6.2, and a Proxmox server. My desktop Windows 10 PC is plugged into a
smaller UPS.

The UPS has a UPS connection to the NAS, and on a power event it shuts the
NAS down properly.

I have Proxmox running on a laptop plugged into the UPS, so it's got its own battery and the APC UPS.

If the power goes out, the synology shuts down, and I can either hit Proxmox on the web interface, ssh into it and shut it down, or hit the power button. If it runs down the battery, it goes into sleep mode.

I'm planning on replacing the laptop with a desktop, and realized that I hadn't thought thruogh the shutdown process. I want to run it headless, so won't have a keyboard and monitor like with the laptop.

I'm wondering if there's a way to have either the Proxmox system or the Synology system connected to the UPS and shut down the other one?


... Are there sections? Consider transitions
--- MultiMail/DOS v0.52
* Origin: realitycheckBBS.org -- information is power. (618:300/1)

Re: UPS homelab setup
By: Kurt Weiske to All on Mon Sep 26 2022 10:05:00

I have a homelab with an APC BackUPS 1500 XM, a Synology NAS running DSM 6.2, and a Proxmox server. My desktop Windows 10 PC is plugged into a smaller UPS.

<CUT>

If the power goes out, the synology shuts down, and I can either hit Proxmox on the web interface, ssh into it and shut it
down, or hit the power button. If it runs down the battery, it goes into sleep mode.

<CUT>

I'm wondering if there's a way to have either the Proxmox system or the Synology system connected to the UPS and shut down
the other one?

exact same ups i have.. and very similar setup except i have 4 servers all on the same ups

run the UPS usb port into the proxmox host and run N.U.T (networked ups thing). then run a nut client on the other machines.

when the thing goes into battery it signals the proxmox machine to shutdown and because the other machines are connected to the nut server on the proxmox machine they shut down too.

https://networkupstools.org/


regards
===

Charles Blackburn
The F.B.O BBS 21:1/221 618:250/36
bbs.thefbo.us IPV4/V6
DOVE-Net FSX-Net MicroNET USENET
--- SBBSecho 3.15-Linux
* Origin: The FBO BBS - bbs.thefbo.us (618:250/36)

I'm wondering if there's a way to have either the Proxmox system or the Synology system connected to the UPS and shut down the other one?

I think APC makes software that does that, but likely wants to charge for it. One UPS power failure can trigger an event to do multiple things.

I never got that fancy with my setup and I have (10) APC UPS units over here. If my power it out long enough for them to go out, everything just powers off immediately. I would try and do a clean shutdown, but even if it goes down hard I have not had issues with my ESXi server/VMs. Worst case, I could restore things.

- Mark

--- WWIVToss v.1.52
* Origin: http://www.weather-station.org * Bel Air, MD -USA (618:100/12.0)

run the UPS usb port into the proxmox host and run N.U.T (networked ups thing). then run a nut client on the other machines.

Is there an equivalent for those using APC UPS using the network management cards (IP based)? I monitor all my APC UPS units using the IP based management cards.

I typically get around 20 or so minutes of runtime in the event of a power outage on most of my stuff, which isn't too bad.

- Mark

--- WWIVToss v.1.52
* Origin: http://www.weather-station.org * Bel Air, MD -USA (618:100/12.0)

Mark Hofmann wrote to Kurt Weiske <=-

I'm wondering if there's a way to have either the Proxmox system or the Synology system connected to the UPS and shut down the other one?

I think APC makes software that does that, but likely wants to charge
for it. One UPS power failure can trigger an event to do multiple
things.

Charles mentioned Network UPS Tools, it appears that Synology supports it -
in fact, it may be using it already for its own network shutdown feature -
one Synology can shut down multiple units.

With that, I can leave the UPS plugged into Synology and run the nut-client
on my Proxmox server.

I never got that fancy with my setup and I have (10) APC UPS units over here. If my power it out long enough for them to go out, everything
just powers off immediately. I would try and do a clean shutdown, but even if it goes down hard I have not had issues with my ESXi
server/VMs. Worst case, I could restore things.

Hey, if you have 10 UPSes, why not? :)


... Make what's perfect more human
--- MultiMail/DOS v0.52
* Origin: realitycheckBBS.org -- information is power. (618:300/1)

Mark Hofmann wrote to Charles Blackburn <=-

I typically get around 20 or so minutes of runtime in the event of a
power outage on most of my stuff, which isn't too bad.

With my current Proxmox + Synology + router setup, I get around 65 minutes
of run time -- that's with two printers plugged into the UPS because I just plugged the UPS from the wall into the UPS.

I'm looking to replace my proxmox laptop with a new SFF desktop I bought, which prompted looking into network UPS support. The shame is that my laptop CPU only draws 15w, the desktop 65w. I'll get less run time out of the UPS with the new system.


... Make what's perfect more human
--- MultiMail/DOS v0.52
* Origin: realitycheckBBS.org -- information is power. (618:300/1)

Re: Re: UPS homelab setup
By: Mark Hofmann to Kurt Weiske on Tue Sep 27 2022 07:25:52

I'm wondering if there's a way to have either the Proxmox system or the Synology system connected to the UPS and shut
down the other one?

I think APC makes software that does that, but likely wants to charge for it. One UPS power failure can trigger an event to
do multiple things.

no need.. networked ups tools is free and is in most package managers... works great...

worked awesome here yesterday with this stupid hurricane.... power went out.. ran on ups for 15 minutes and then shut everything down except my cablemodem and firewall (on a separate ups as they dont draw much and can stay up for hours)

I was more pissed off that we had it up until 11am yesterday and after the thing went through then didnt come back till 3am this morning....

regards
===

Charles Blackburn
The F.B.O BBS 21:1/221 618:250/36
bbs.thefbo.us IPV4/V6
DOVE-Net FSX-Net MicroNET USENET
--- SBBSecho 3.15-Linux
* Origin: The FBO BBS - bbs.thefbo.us (618:250/36)

Re: Re: UPS homelab setup
By: Mark Hofmann to Charles Blackburn on Tue Sep 27 2022 07:28:30

run the UPS usb port into the proxmox host and run N.U.T (networked ups thing). then run a nut client on the other
machines.

Is there an equivalent for those using APC UPS using the network management cards (IP based)? I monitor all my APC UPS units
using the IP based management cards.
I typically get around 20 or so minutes of runtime in the event of a power outage on most of my stuff, which isn't too bad.

check the wiki for nut, i know it supports some form of IP management but as mine are all usb based i didnt look.

but i have a mix of small 800kva apcs for things like the tvs etc, and then a big 1500kva cyberpower for the 4 servers

regards
Z
===

Charles Blackburn
The F.B.O BBS 21:1/221 618:250/36
bbs.thefbo.us IPV4/V6
DOVE-Net FSX-Net MicroNET USENET
--- SBBSecho 3.15-Linux
* Origin: The FBO BBS - bbs.thefbo.us (618:250/36)

Re: Re: UPS homelab setup
By: Kurt Weiske to Mark Hofmann on Fri Sep 30 2022 06:39:00

I never got that fancy with my setup and I have (10) APC UPS units over here. If my power it out long enough for them
to go out, everything
just powers off immediately. I would try and do a clean shutdown, but even if it goes down hard I have not had issues
with my ESXi server/VMs. Worst case, I could restore things.

Hey, if you have 10 UPSes, why not? :)

i dont see why you dont just daisy chain them at that point lol

regards
===

Charles Blackburn
The F.B.O BBS 21:1/221 618:250/36
bbs.thefbo.us IPV4/V6
DOVE-Net FSX-Net MicroNET USENET
--- SBBSecho 3.15-Linux
* Origin: The FBO BBS - bbs.thefbo.us (618:250/36)

Re: Re: UPS homelab setup
By: Kurt Weiske to Mark Hofmann on Fri Sep 30 2022 07:21:00

I'm looking to replace my proxmox laptop with a new SFF desktop I bought, which prompted looking into network UPS support.
The shame is that my laptop CPU only draws 15w, the desktop 65w. I'll get less run time out of the UPS with the new system.

dont forget though, it's not pulling 65w all the time unless it's under heavy load

regards
===

Charles Blackburn
The F.B.O BBS 21:1/221 618:250/36
bbs.thefbo.us IPV4/V6
DOVE-Net FSX-Net MicroNET USENET
--- SBBSecho 3.15-Linux
* Origin: The FBO BBS - bbs.thefbo.us (618:250/36)

I never got that fancy with my setup and I have (10) APC UPS units

over MH> here. If my power it out long enough for them to go out, everything MH> just powers off immediately. I would try and do a clean shutdown, but

even if it goes down hard I have not had issues with my ESXi server/VMs. Worst case, I could restore things.

Hey, if you have 10 UPSes, why not? :)

At one point I was running an official APC VM appliance that listened to my network managed APC units and I could have them perform shutdowns of my VMs if there was no power after xx minutes.

At the time, most of our power related issues were not that long. Often times the shutdown would start and my power would be back on, so it ended up creating me more work to bring everything back up for nothing. It just didn't work as well as I had liked and all my UPS last different amounts of times based on load.

I found doing the manual shutdown if I'm around worked better, but certainly am not opposed to giving it another shot with different software and options. There is likely something better these days that would work with my ESXi setup and Windows servers/systems.

- Mark

--- WWIVToss v.1.52
* Origin: http://www.weather-station.org * Bel Air, MD -USA (618:100/12.0)

With my current Proxmox + Synology + router setup, I get around 65
minutes of run time -- that's with two printers plugged into the UPS because I just plugged the UPS from the wall into the UPS.

Wow, that is decent runtime but the load is way less than my setup. I have (2) APC 1000s going to each of my SuperMicro servers - one to each power supply. I get around 15-20 minutes of runtime, which is pretty good considering these use around 350 watts on average at least.

My workstation is a power hog since I have (5) GPUs on it. I have an APC 1000 and APC 1500 on it and get around 10 minutes runtime. The 5 displays are on the APC 1000 and the PC is on the 1500. The PC uses around 850 watts on average - which is more than both my Supermicros use added together even with 30 hard drives in the one.

- Mark

--- WWIVToss v.1.52
* Origin: http://www.weather-station.org * Bel Air, MD -USA (618:100/12.0)

worked awesome here yesterday with this stupid hurricane.... power went out.. ran on ups for 15 minutes and then shut everything down except my cablemodem and firewall (on a separate ups as they dont draw much and can stay up for hours)

It appears there is a NUT port for Windows and I knew it also works with Home Assistant (which I also use at home).

I'll take a look and see how it works. Hopefully there is a web gui and I can see all the UPS units in the house via a central place.

I do monitor them in Home Assistant using the regular snmp commands to poll the different information.

- Mark

--- WWIVToss v.1.52
* Origin: http://www.weather-station.org * Bel Air, MD -USA (618:100/12.0)

Re: Re: UPS homelab setup
By: Mark Hofmann to Charles Blackburn on Sat Oct 01 2022 09:45:56

worked awesome here yesterday with this stupid hurricane.... power went out.. ran on ups for 15 minutes and then shut
everything down except my cablemodem and firewall (on a separate ups as they dont draw much and can stay up for hours)

It appears there is a NUT port for Windows and I knew it also works with Home Assistant (which I also use at home).
I'll take a look and see how it works. Hopefully there is a web gui and I can see all the UPS units in the house via a
central place.
I do monitor them in Home Assistant using the regular snmp commands to poll the different information.

there is one for windowws, but getting it to work right is a pain and i gave up on it.

i beleive you can use SNMP to pull the mibs, but i havent gotten that far to be honest.. one of those things i picked up
and then put down to do something else lol

regards
===

Charles Blackburn
The F.B.O BBS 21:1/221 618:250/36
bbs.thefbo.us IPV4/V6
DOVE-Net FSX-Net MicroNET USENET
--- SBBSecho 3.15-Linux
* Origin: The FBO BBS - bbs.thefbo.us (618:250/36)

On 01 Oct 22 09:37:05, Mark Hofmann said the following to Kurt Weiske:

I found doing the manual shutdown if I'm around worked better, but certainl am not opposed to giving it another shot with different software and option There is likely something better these days that would work with my ESXi se and Windows servers/systems.

As I run my own Email server... I take the UPS-emails generated from the APC daughtercard when power goes to battery-state and execute a simple DOS batchfile which just creates a dummy file in an SMB share.

All of the VM's check every 15 minutes to half-hour for the presence of that dummy file. If it exists, they shutdown. Call it a simple "heartbeat" event.

That checking also goes along with an automated disk-space check via a crude commandline utility I wrote. If space falls below a certain threshold, a dummy file is created. If the batchfile sees that dummy, I'm sent an email alert.

Same with any processes that suddenly may stop running... ie. BinkD on the
BBS etc. I wrote a simple program that checks for any running process I specify. If it doesn't exist, another dummy tells the batchfile to send email alerts.

No powershell, WMI, 3rdparty stuff, SNMP or other nonsense in any of this. Just simple DOS commands... thus the CPU/resources are never spiked.

Nick

--- Renegade vY2Ka2
* Origin: Joey, do you like movies about gladiators? (618:500/24)

i beleive you can use SNMP to pull the mibs, but i havent gotten that far to be honest.. one of those things i picked up
and then put down to do something else lol

That is what I have done to pull the information into HomeAssistant and other 3rd party programs. I find the mibs that I'm interested in using one of those snmp mib browsers, and then just use that information to pull in the data.

- Mark

--- WWIVToss v.1.52
* Origin: http://www.weather-station.org * Bel Air, MD -USA (618:100/12.0)

Mark Hofmann wrote to Kurt Weiske <=-

With my current Proxmox + Synology + router setup, I get around 65
minutes of run time -- that's with two printers plugged into the UPS because I just plugged the UPS from the wall into the UPS.

Wow, that is decent runtime but the load is way less than my setup.

Way Way less. Mine's more of a "spin up and test, then shut it down"
homelab, the only loads I have running full-time are the BBS, a Windows DC
and PiHole.

Theoretically, I could get even longer run time by taking my Proxmox server off of the UPS and use NUT to shut it down, since it's a laptop with a battery. It comes in handy - when I had an extended power outage, I brought the laptop to work, connected to guest wifi and transferred mail packets
from my uplink to my hosts and vice versa.


... THE SEVEN JOURNEYS TO ITSELFNESS
--- MultiMail/DOS v0.52
* Origin: realitycheckBBS.org -- information is power. (618:300/1)

As I run my own Email server... I take the UPS-emails generated from the APC daughtercard when power goes to battery-state and execute a simple
DOS batchfile which just creates a dummy file in an SMB share.

All of the VM's check every 15 minutes to half-hour for the presence of that dummy file. If it exists, they shutdown. Call it a simple
"heartbeat" event.

That is very creative! I actually do something similar to kickoff mail tossing events for my classic WWIV BBS using a Windows program called AutoMate. You can create events to check for a file to exist in a directory, modified file, etc, and then kickoff batch files. It all works really well. Never came to mind that a similar process could be used for UPS/system shutdowns.

For ESXi, I believe there are command line actions that would perform shutdowns that could be added to batch files. Mainly because not all my VMs are Windows, they are all sorts of stuff that I would want to have ESXi perform a graceful shutdown.

Then the issue becomes the power-on when power is restored. Mainly because I would need at least one of my domain controllers to come up first or some other systems would boot up and have issues and need to be rebooted again.

Of course I could just do all this manually like I do today in the event power it out for an extended time. That has not happened here for years, fortunatly.
I do remember running my ESXi cluster and everything all 100% from a gasoline generator after we had a hurricane when the power was out for two days. I was back feeding the entire house from the gas generator and was able to run all types of stuff. Microwave, furnace, servers, refrigerators, etc. All on a 5k watt unit. It stuggled at times and the lights would dim sometimes, but it worked.

- Mark

--- WWIVToss v.1.52
* Origin: http://www.weather-station.org * Bel Air, MD -USA (618:100/12.0)

I forgot to mention this bit about the home email server. Technically I sort of run my own email server, but it is really just a relay these days. I have it send all mail to my free Mailjet account for delivery.

I do have a powered off VM that has Exchange on it. If Google was going to start charging me, I was ready to dump them. That didn't happen so I left things alone.

- Mark

--- WWIVToss v.1.52
* Origin: http://www.weather-station.org * Bel Air, MD -USA (618:100/12.0)

Theoretically, I could get even longer run time by taking my Proxmox
server off of the UPS and use NUT to shut it down, since it's a laptop
with a battery. It comes in handy - when I had an extended power outage,
I brought the laptop to work, connected to guest wifi and transferred
mail packets from my uplink to my hosts and vice versa.

Sounds like you have double battery backup. The UPS and the laptop battery, itself! :)

That is an interesting idea. Maybe I should start daisy chaining my APC UPS to get longer runtime.

- Mark

--- WWIVToss v.1.52
* Origin: http://www.weather-station.org * Bel Air, MD -USA (618:100/12.0)

On 07 Oct 22 18:08:09, Mark Hofmann said the following to Nick Andre:

I forgot to mention this bit about the home email server. Technically I so of run my own email server, but it is really just a relay these days. I ha it send all mail to my free Mailjet account for delivery.

I've ran my own email system for at least 15... 20 years now I think.

I run Hmailserver as an SMTP front-end for my Exchange server.

The only work I've ever had to do in that timeframe was configure DNSBL correctly in the beginning and later on a few Txt records in DNS for some weird new fad that basically told the world "please don't spam me".

I dunno, I guess since I work intimately with Exchange and Outlook in my career, why not run it at home. Once you disable all the debugging and other crap that comes out of the box its really not that bad, takes up little resources here. I don't like building new VM's every time a new Exchange version comes out but its not like that happens often.

In all this time I've never had a problem delivering email to anyone and have no incentive to stop as I have a commercial grade fiber Internet connection.

Nick

--- Renegade vY2Ka2
* Origin: Joey, do you like movies about gladiators? (618:500/24)

On 07 Oct 22 18:04:34, Mark Hofmann said the following to Nick Andre:

For ESXi, I believe there are command line actions that would perform shutdowns that could be added to batch files. Mainly because not all my VM are Windows, they are all sorts of stuff that I would want to have ESXi perform a graceful shutdown.

I never really got into the Exsi-specific commands, because as explained once all the VM's sense that dummy file they shutdown via. the Windows shutdown command. I'm not running non-Windows stuff here though...

Then the issue becomes the power-on when power is restored. Mainly because would need at least one of my domain controllers to come up first or some other systems would boot up and have issues and need to be rebooted again.

What about the startup-sequence settings in Exsi? List the DC first on that one server and just use timer sequence on the others?

Of course I could just do all this manually like I do today in the event po it out for an extended time. That has not happened here for years, fortunatly.

I have one APC SmartUps 1400 and replace the batteries every couple of years from some local vendor that actually sells them at a decent price. The daughterboards for those APC UPS's are a godsend and I'll never understand why Ethernet or probe-contacts were never made "standard issue" for their stuff until much later.

I've actually never had a full-cycle on UPS, I'm guessing I'll get about a half hour or so of runtime with all the stuff in the rack. Non-essential stuff just goes to the main PDU... I don't need NAS storage during an outage.

Nick

--- Renegade vY2Ka2
* Origin: Joey, do you like movies about gladiators? (618:500/24)

I've ran my own email system for at least 15... 20 years now I think.

I run Hmailserver as an SMTP front-end for my Exchange server.

Nice! I never heard of Hmailserver until just now. I was using the full blown MDaemon for hosting my own email years ago. Now, I just use it for some SMTP relay for certain things at home - mostly alerts.

The only work I've ever had to do in that timeframe was configure DNSBL correctly in the beginning and later on a few Txt records in DNS for some weird new fad that basically told the world "please don't spam me".

Yeah, probably SPF records and/or DKIM. For some reason when you email me, Google has a warning on the top that says "Be careful with this message". Weather-station.org mail could not verify that is actually came from net229.org.

I have seen that before from other senders. I had to do some magic with SFP and DKIM records so that Mailjet could act as my external SMTP relay for my domain without getting those type of warnings messages.

I dunno, I guess since I work intimately with Exchange and Outlook in my career, why not run it at home. Once you disable all the debugging and other crap that comes out of the box its really not that bad, takes up little resources here. I don't like building new VM's every time a new Exchange version comes out but its not like that happens often.

I still have my Exchange VM local all set up and ready to go, just never switched over. I was seriously considering it when Google was trying to exploit business rate funds from personal home users to use GSuite.

In all this time I've never had a problem delivering email to anyone and have
no incentive to stop as I have a commercial grade fiber Internet connection.

Nice! I don't have business service so my IP is technically DHCP even though it never changes unless I swap my external firewall (ethernet MAC changes). I use a dynamic DNS updater just in case that uses an API to Cloudflare to update my DNS records in that rare case it changes.

- Mark

--- WWIVToss v.1.52
* Origin: http://www.weather-station.org * Bel Air, MD -USA (618:100/12.0)

I never really got into the Exsi-specific commands, because as explained once
all the VM's sense that dummy file they shutdown via. the Windows
shutdown command. I'm not running non-Windows stuff here though...

I have a mixture of all types of stuff running as VMs. Windows, various Linux, some Linux based appliances, etc. Technically, vCenter itself is one of those appliances.

What about the startup-sequence settings in Exsi? List the DC first on
that one server and just use timer sequence on the others?

Yes, there is a startup delay that can be added in ESXi to auto-start VMs in a certain order. It seemed to work ok the few times I tried it, just have to tweak the timers to get the right settings.

I have one APC SmartUps 1400 and replace the batteries every couple of years from some local vendor that actually sells them at a decent price. The daughterboards for those APC UPS's are a godsend and I'll never understand why Ethernet or probe-contacts were never made "standard
issue" for their stuff until much later.

I've actually never had a full-cycle on UPS, I'm guessing I'll get about
a half hour or so of runtime with all the stuff in the rack.
Non-essential stuff just goes to the main PDU... I don't need NAS storage during an outage.

I order all my APC batteries from Battery Sharks. They have the best price online and I can't find any better deals locally. I typically get around 3 years on them before they give me the error that they need to be replaced. SInce I have so many of them, I seem to be always replacing one of them. I just swapped the batteries in the one that runs my main PC just last week.

- Mark

--- WWIVToss v.1.52
* Origin: http://www.weather-station.org * Bel Air, MD -USA (618:100/12.0)