Friday, August 9, 2013

Disabling Data Execution Prevention (DEP) for Windows 7 - Part I

This is a general research notes on the new way Date Execution Prevention (DEP) in Windows 7 systems prevents many of the old applications getting executed on the system (mostly memory address areas). There are various details that can be looked into for in-depth knowledge on this issue. Truly speaking this is tight and there is no such workaround except ROP gadgets (Return - Oriented Programming) that is more of ethical hacking than a workaround and is a subject in itself.
The problem which usually comes up with is more technical and is related to diverse factors.

Actual Exception from Visual Studio:
This actual exception comes up while instantiating the new Java Axbridge ActiveX object.
System.AccessViolationException was caught Message="Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
This followed up by NullReference exception as it failed to create the object.
Standalone Executable may come up with the following, though it points to the same:
System.Runtime.InteropServices.COMException (0x80040154):
Retrieving the COM class factory for component with CLSID {D824B185-AE3C-11D6-ABF5-00B0D07B8582} failed due to the following error: 80040154. 

Research Areas:
1) 32 bit/64 bit application compatibility on Windows 7 (64 bit) system.
2) Oracle Java JRE support for Windows 7 system (since Java axbridge components were used).
3) Microsoft .NET component supports for 64 bit systems.
4) The concepts related to PAE (Physical Address Extension) is worth mentioning.

This post is an accumulation of important points and references of various links, that helped resolving the actual problem.

What was tried that did not help:

It is more or less pointing to compatibility issue where all the following did not work in our case:
  • Recompilation in 32 bit mode (both client and server that includes the Java interoperability DLL).
  • We verified that DEP (Data Execution Prevention) which is by default turned ON for all applications in Windows 7 does not actually contribute to this. Trying to enlist the application as a DEP exception did not help as Windows 7 did not allow that.
  • Re registering the components in Windows 7 for 64 bit.
  • Force converted the DLL to 32 bit using Corflags.exe (did that today forgot to mention).
  • Dump file analysis.
  • We copied the same executable from the Windows 7 machine to the Windows XP machine and upon execution there was no execution.

There are various issues with AxBridge Java ActiveX DLL gets invoked from 64 bits machines. These were also investigated in details. I have kept this article brief and is a part of the development/troubleshooting process.
The next part of this article will be released soon that talks about an indepth analysis and the actual fix.