1.Is there any way to filter packets by the sender host name?
Not dynamically ... as in: here comes a packet from random-never-seen-before source IP address, check if it's from "somedomain.com". Some setup commands allow you to enter FQDN instead of IP address and that rule then "dynamically" changes ... if FQDN resolves into another IP ... but it only does it after DNS TTL expires.
The fact is that firewall is (more or less) L3 stuff and that operates on IP addresses.
2.How to limit communication for only the hosts that are joined to a specific domain?
See my answer above. If you can construct an address list containing IP addresses of all hosts from a specific domain, then you could construct firewall filter rule using that address list. You can not use domain name as constructor of the list, because usual DNS clients can't get whole list from DNS servers (that would be zone transfer and most DNS servers are configured such that zone transfers are only allowed for certain DNS servers, most notably secondary DNS servers for same domain).
Another way (not hugely practical) is to construct L7 filter rules. The problem with L7 rules is that they are very CPU intensive and that it's only possible to use them for a few particular L5 protocols (e.g. http or https) where destination server name
is mentioned early in the connection.
3.how to integerat mikrotik and active directory and set firewall rule based on domain name of clients?
AFAIK ROS doesn't talk LDAP (or AD or whatever), you'd have to use a Radius server which would translate LDAP/AD policies into Radius policies.