Restrict OpenVPN clients based on source IP

When enabling “Remote User Access” in a router (i.g. OpenVPN), a potential vulnerability is enabled. Is there a way to restrict the inbound access by defining a firewall rule to restrict the source of the logins? The logical place would be in the “internal services” rules section, but no rule can be setup to only allow access from a specific address.
Has anyone got a solution to this?