· There are of course other possible ways to tackle the problem. For instance, when Ida breaks at your int3, try to trace back to find from where the code sequence containing the int3 is called. This should be connected to the anti-debug code, because I assume your int3 is not called when running without debugger.
· I've read that the INT 3 (0xCC) is used for software breakpoints. It is set by (for instance) a debugger by overwriting the actual program code in memory. I've also read that INT 3 is a "trap" not "fault" exception meaning the address pushed on the stack …
· Replace the first byte at the target address with the int 3 instruction; Then, when the debugger asks the OS to run the process (with PTRACE_CONT as we saw in the previous article), the process will run and eventually hit upon the int 3, where it will stop and the OS will send it a signal.
· In some cases the developers (during development) may put explicit INT 3 instructions in the code they are developing, knowing that if that condition happens, their program will *break* and the debugger they are using will take the charge. When active the debugger sets his own interrupt handler for INT3 in the system.
Int 3 is used to trigger a breakpoint. The interrupt handler is tiny, and neither the interrupt nor its handler stop any threads. If there is no debugger loaded the handler will either ignore it or call the OS to take some kind of error action like raising a signal (perhaps SIGTRAP).
1 Answer1. Active Oldest Votes. 2. It is setting a trap flag with that xor instruction. when it is run normally (not under debugger) the trap flag is triggered so the handler gets a chance to execute. when the binary is run under debugger the trap flag is ignored and the handler doesn't get a chance to execute.