I can imagine... When my XF was running poorly, I drove it up the street to test, and it barely got out of it's own way.   I'd suggest to get it running nicely first, then decide. It's a big expensive undertaking to swap in other drivelines and have it daily-driver reliable. (especially these days, with rising costs, etc)   It should be brisk and torquey, but not certainly 'fast' like a modern vehicle. World's cheaper/easier to diagnose/maintain and repair though, and I'd choose that any day.  

From ur video angle, it looks like no.1, no.4 and no.5 intake valves aren't opening correctly. Either partly bent pushrod/s (remove to check) collapsed/stuck lifter, or worn camshaft.   I'd also see what the vacuum gauge reads, now that loose pushrod is fixed.   If u have a lifter removal tool, with some maneuvering, u can remove the lifters without undoing the cylinder head. Same for the camshaft. The radiator/AC condenser needs removing to get the camshaft out though.   With my XF wagon, I removed each lifter and disassembled/cleaned/refitted them. I had x3 bent pushrods, all on the intake side.          

Around 20-25nm I believe

I can't remember what gearbox you have?  if it's auto, then there's column shift V8 stuff that would fit up to it the same.    if it's manual, well you'll be unlikely to get it working (not sure i've seen a falcon with 3 speed V8 manual)
 

Yeah mines much slower than that. I forgot to mention that the bolt was backed out and someone else had been in there as evidenced by all the bolts being finger tight and the rear one being gone so I'm wondering if someone's been at it before

i had seen some XFs with the 2 different crossmember spots, likely the leaded ones but not idea to be honest.    the lifters when NEW have a dome shape on the bit that runs on the cam ( i have never seen this on any used ones) when they are normal they'll be fairly flat(they wear in on cam run in). when they are stuffed they are dished inwards, and when the cam is wiped they might not even have an even base but they'll not be looking good. (pull the one out next to it that is working better, and compare.. )    google has some pics on this link    
this was My stock standard XF ute column auto one, ZL fairlane cluster fitted.  probably on LPG.     

I heard somewhere that the XF 1 (what I got) still had the xe crossmember although I'm not sure how I would hook up the column shifter. Is there anything wrong with using the stock box or won't it fit? Also with the lifter do I just look for deep marks or is there something else to look out for? I really like the XF but it's just so gutless I don't know if it's just because of mine running crapily or if it's just an XF thing but when I put my foot down it takes almost 30 seconds to hit 100 which just seems too slow. What was yours like? 

what i'd do is grab the $150 one below and slap it in pesonally, gets it going easy. fix the original later maybe.
351 into XF, bit of mucking around but i've done it in a weekend when i had all the bits ready. 
the heater pipes foul on the engine/heads so there's a couple of ways to get around that.    the gearbox crossmember on XF only has one position unlike XD/XE,  you might need to mod the cross member to suit depending on gearbox choice (a 351 and FMX auto from 1977LTD dropped straight into My brothers with the heater pipes bent up (they cracked the solder, so this needs doing with heater core removed) and some hammering of the firewall.    used XE V8 accel cable on the thermoquad carby, but if you have a holley you can use the 6cyl cable from the back side instead of pulling from the front.     

Also how hard is it to pull the cam outta one of these. If that's my issue I could get a used cam pretty easy.

There's one near me for $150 but at the same time there's also a 351c for sale near me for $800 so you can see my dilemma. The only reason I wanted to keep the Crossy is because I thought this car was a survivor. However with the rust and dodgy repairs on it the rose tint is starting to wear off. Honestly it all depends how much is required to put a clevo in it. If it's not gonna take forever then I'll probably do that (I got the legal side covered allready so it's just mechanical). However if it isn't gonna take too much to fix the Crossy then I'll keep it as I do like to keep things original. What are your thoughts?