Without a doubt Romans 5:12-18 is the most misunderstood passages of the New Testament and a false understanding of this passage provides the very foundation for the myth that Adam's spiritual death was imputed to all his descendants. Let us look first at Romans 5:12:
"Wherefore, as by one man sin entered into the world, and death by sin; and so death passed upon all men, for that all have sinned" (Ro.5:12).
From this we can understand the following: (1) Sin entered the world when Adam sinned and that sin brought about spiritual death. (2) Adam's sin was somehow responsible for bringing spiritual death to all men. (3) This death came to all men because all have sinned.
What this verse does not tell us is exactly "how" Adam was responsible for bring death to all men.
Reverting back to your OP . . .
The answer to your question rests in the truth that Adam was created as representative and head of the entire human race. All persons born trace back to him. So when Adam sinned, his guilt was imputed to all his offspring, and the sentence for that guilt is death. Romans 6:23
You argue that men are declared guilty for their own sins, not Adam's and you are correct. All of are sinners for both reasons. First, because Adam corrupted our nature, and second because subsequently, we are all sinners.
Romans 5:14 specifically says that death reigned over all,
even over those who had not sinned according to the likeness of Adam. . .
What was the difference between Adam's sin and the sins of all men?
It is the fact that Adam alone sinned as federal head of the human race, and the rest of mankind sins as a result of being imputed with his guilt.
This is important to see, in order for us to comprehend the federal headship and representation of righteousness provided to mankind by the last Adam, Jesus Christ. Through His righteousness and obedience, grace abounds for all those He represented in His death and resurrection.
" . .
Adam, who was a type of Him who was to come. . . by the grace of the one Man Jesus Christ abounded to many." Romans 5:14b-15
"Federal Headship" is vital to understanding the necessity of the Incarnation, death, and resurrection of Jesus Christ as well as why sin has reigned over all men.