Using a 64 bit OS means that there will more pressure on RAM and possibly disk space as code is somewhat larger. Also, effective cache size is slightly less since pointers will take up more space. Normally, I would recommend more RAM and that means that swap should also grow. If you have 64-bit applications or database then, usually the positives greatly outweigh any negatives.
If you are concerned about performance, then the first step is to understand performance in your current environment. If you are I/O bound and don't do any other adjustments, a 64-bit OS likely won't solve your problems. Since the 64-bit OS can run 32-bit applications, there are few reasons to use the 32-bit OS on 64-bit hardware, if RAM and disk are not limiters.
Mom 6