On macOS, you can use the Open Scripting Architecture (OSA) with JMRI to send instructions to OS X or other applications.
You can embed or call OSA scripts within JMRI scripts, invoking them to have your Mac perform functions outside JMRI. An example of this is in the AppleScript.py sample script distributed with JMRI. It executes a simple AppleScript:
tell application "Finder" make new folder at desktop end tell
The sample script shows how to do this, and can easily form the basis for running your own OSA scripts from within JMRI.
JMRI 4.5.2 or newer can directly call AppleScript without embedding it in another script by adding JMRI AppleScript Support to your JMRI installation. Note that this may not work in future versions of macOS or Java.
For more information on AppleScript, please see:
A Google search will turn up lots of interesting resources.