A user's access to a folder is controlled by their lowest level of access, whether from a group or their individual user record. For example:
User N is in Groups A and B.
Group A has full access to the folder but group B cannot Add documents.
Therefore, user N will not be able to Add documents based on the permission set for group B.
Changing the permissions of the Public Group affects all users, as everyone is in this group and cannot be removed. Remove Public Group from the folder permissions area entirely rather than changing its permissions — otherwise all users may lose access. If a user is denied View access within folder permissions (whether by group or individual setting), they will not be able to interact with documents in that folder at all.