On Windows Server Core without desktop experience feature installed, you should run the installer by remote desktop under DOS command or Powershell, do not run the installer under Windows Admin Center -> Powershell remotely.
DKIM installer supports silent mode, you can use the following command to start silent installation.
start /wait eaxchdomainkeys.exe /S
start eaexchdomainkeys.exe -ArgumentList /S -Wait
The following arguments are case-sensitive.
Because DKIM Gui Manager doesn't support server core without desktop experience installed,
so you can use "DKIM Management Shell" to manage DKIM configuration.
DKIM Management Shell
After you installed DKIM on Server Core + Exchange Server, but DKIM manager is not available.
You can run DKIM installer on a client machine (Windows 7 - Windows 10) without Exchange Server installed, the DKIM management tool will be installed.
After the DKIM manager is installed, open DKIM manager and create domain(s), then click "Open Config Folder", find [domain].json files in config folder, copy those files to "Your Exchange Server\DKIM installation path\dkim", then your Exchange Server has the same configuration.
See Also
Using DomainKeys/DKIM
Deploy Public Key in DNS server
Test DomainKeys/DKIM signature
Troubleshooting
Using Selector
Using Sender Rule
Appendix - Set up SPF record in DNS server
Appendix - Set up DMARC record
Appendix - DKIM/SPF/DMARC Inbound Authentication in Exchange Server
Appendix - Use DkimPowerShell Module in PowerShell