In SharePoint Online modern experience, if you create SharePoint Online Team site it also create Office 365 group associated with it. In addition, three SharePoint groups like Owners, Members and Visitors created and if you try to change the permissions of these groups, you will find the "Edit User Permission" or "Remove User Permission" grayed out.
By default SharePoint associate these groups to Office 365 membership and use these groups to manage the owners, members and visitors of office 365 group membership as well. Hence you can not change the permission of these groups manually.
There are some alternative options like you can change the SharePoint Online Team site default/associated groups so that you can use the existing groups with required permission. In order to change the association, you can either create new SharePoint groups or select the existing groups by navigating below SPO URL.
https://<YOUR DOMAIN>.sharepoint.com/sites/<SITE NAME>/_layouts/15/permsetup.aspx