The hand that held my wrist Was battered on one knuckle;
The father is grasping the son's wrist. Notice that this is different from the way the father's referred to in the rest of the poem; his hand is referred to as "the" hand, not "your" hand.
The hand is battered, but only on one knuckle. We don't quite know what "battered" means. It could just be cracked and dry from a long hard day of work, but then wouldn't every knuckle be battered? Maybe the father whacked it on something – perhaps he punched something, maybe he was carelessly drunk, or maybe he scraped it at work. We don't know.
Lines 11-12
At every step you missed My right ear scraped a buckle.
We get a little more about how this father-son relationship could be a little violent. Every time the father misses a step in his waltz, the son's ear scrapes against his belt buckle. The father seems to be gallivanting along, totally unaware that he's scratching his little boy as he goes.
A belt buckle also has a violent connotation, because in the past it was common for fathers to use their belts to beat their children (this still happens sometimes today).
We also see that this boy is very small – it seems like he's only as tall as his father's belt.