Example of error message. Note that 'ThumbnailHandler.dll' isn't a required file, you can still use DSM without it. In DSM 4.0, you may see an option to continue the install with this error which you should go ahead with.
In case of no option to continue the install, here's a fix:
Turn off Data Execution Prevention (DEP) temporarily before installing DSM 4.0. After install is complete, re-enable it.
a. Right-click on Start button and select Command Prompt (Admin). If you only see PowerShell, type in CMD from Windows search and right-click to open as administrator.
b. Once the command prompt is open, try these steps to disable the DEP, complete DSM setup and re-enable the DEP.
c. Click on Enter after typing each command.
bcdedit.exe /set nx AlwaysOff
d. Run DSM 4.0 setup (both main release + translators) till completed.
d. To enable DEP back, type the command and hit Enter.
bcdedit.exe /set nx AlwaysOn
e. Close command prompt.
May still work for some users:
To fix the issue, please go into Windows 'Control Panel' -> 'System' ->'Advanced system settings' . Click on performance settings and go the 'Data Execution Prevention' tab.
Choose 'turn on DEP for all programs except..' and then add the DesignSpark Mechanical 'setup.exe' installer file.
Click 'apply -> ok' and then run the DS Mechanical software installer.