Yes your impression is correct, 32-bit architecture is limited to addressing a maximum of 4 gigabytes of memory. Depending on the operating system, this number can be cut down even further due to reserved address space etc.
However with the use of the use of PAE (Physical Address Extension if supported by the processor) we can enable addressable memory to more then 4GB but it does not change the size available to a single process/binary i.e. 4GB.