That is correct. But that death only happens when men sin:
"Wherefore, as by one man sin entered into the world, and death by sin; and so death passed upon all men, for that all have sinned" (Ro.5:12).
The next verse gives us a clue as to in what way Adam was responsible:
"...even as by one man sin entered into the world, and by sin death; and thus death passed upon all men, for that all have sinned: for until law sin was in the world; but sin is not put to account when there is no law" (Ro.5:12-13; DBY).
These verses are speaking of "law" in a "universal" sense because the "deaths" being considered are also "universal" in nature: "death passed to all men." The only universal law that has been in effect since Adam is the law which is written in the heart of all men, the same law of which the "conscience" bears witness:
"For when the Gentiles, which have not the law, do by nature the things contained in the law, these, having not the law, are a law unto themselves: Which shew the work of the law written in their hearts, their conscience also bearing witness" (Ro.2:14-15).
When Adam ate of the "tree of the knowledge of good and evil" he had the knowledge of the law written in his heart and his "conscience" bore witness to that law. His very nature had changed. The Lord said:
"Behold,the man is become as one of us, to know good and evil " (Gen.3:22).
Man now had a "conscience" of the law written in his heart. All of Adam's descendants would thereafter be born in Adam's likeness and image, also having a "conscience", or an inborn knowledge of God's law:
"And Adam lived an hundred and thirty years, and begat a son in his own likeness, and after his image; and called his name Seth" (Gen.5:3).
So Adam was responsible for death coming unto all men because he was responsible for bringing "law" unto all men.