Firstly, if ADV riding is a new experience for you then like anything there is a learning curve involved: it's a confidence thing because the bikes you are used to are lighter and lower. You'll get there - think back to when you first started racing...did you ever have an "oh ****" moment back then? - but you learn and become familiar, and gain experience, right?
Height-wise, there are other guys on here who are around your height - I'm a bit taller, but not much, and I have the low seat which you can buy separately. TBH I don't use it that much because the standard seat in the high position gives me a comfortable highway ride and I'll switch it to the low position when I arrive at the trails. However, if I think I'm going to encounter slippery mud I will fit the low seat so that I can get at least one foot firmly down for paddling and stability.
Weight-wise: yeah, it's a pretty heavy bike (although not as heavy as most other big ADV bikes), but it carries its weight really well and most of the time off road you'll be up on the pegs for stability anyway. This bike will descend a steep gravel and scree incline at walking pace no problem, so it's pretty well balanced despite the weight. Practice slow-speed riding up on the pegs and you'll get used to dealing with the weight over time as your confidence builds.
Mate, everyone who rides ADV bikes has had those "oh ****" moments: I had one last Wednesday - honestly! And FWIW, yeah, I did ride dirtbikes when I was much younger, but they were much lighter and for the last 20 years I've been riding sportsbikes, so I know where you are coming from. I'd say get some decent 50/50 tyres fitted (yes, the stock tyres are crap), fit some crash protection, the low seat and a steering damper and give yourself a year on the bike. Take it off road as often as you can, start easy and build your confidence bit by bit - trust me, you'll get there. Oh, and BTW if you head off road you WILL drop the bike at some stage...most of us have...lots of times, but it will usually be an unexpected low-speed stall, so no real harm done, and it's not too bad to pick up if you get the technique right - even easier if you have a mate with you.
Stick with it mate, because ADV-riding is an awesome experience and the AT is an awesome bike to do it on.