heur-detect

virus choices?

• why don’t viruses always append/replace
  • “head/tail” scanning
• why don’t viruses always change start location
  • checking for suspicious start location
• why did I bother talking about all these strategies?

more on virus strategies

• after we talk about anti-virus strategies some

anti-malware software goals

• prevent malware from running
• prevent malware from spreading
• undo the effects of malware

  ---

• key subproblem: detect malware

tripwire

• open source tool from c. 2000
  • also company around that tool, but I don’t know about it
• “tool for monitoring and alerting on file & directory changes”
• targetted at servers with professional administrators

  ---

• setup: run tool, it records state of system/etc. files (e.g. hashes)
• later: run tool, it tells you if anything changed

tripwire as antimalware software?

• tripwire idea: detect any changes + notify sysadmin about them

  ---

• what is sysadmin supposed to do with this info?
• what about normal software updates, etc.?
• can malware hide in files that are supposed to change?
  • “data” files with other programs, scripts?
  • …
• what if system compromised before setup?

modern bootloaders — secure boot

• ‘‘Secure Boot’’ is a common feature of modern bootloaders
• idea: UEFI/BIOS code checks bootloader code, fails if not okay
  • requires user intervention to use not-okay code

Secure Boot and keys

• Secure Boot relies on cryptographic signatures
  • idea: accept only ‘‘legitimate’’ bootloaders
  • legitimate: known authority vouched for them
• user control of their own systems?
  • in theory: can add own keys
• what about changing OS instead of bootloader?
  • bootloader could check cryptographic signature or hash of kernel being loaded

malware “signatures”

• typically can’t rely on whitelisting approach
  • software and related files change legitimately
  • and malware might not be in main executables

    ---

• antivirus vendor have signatures for known malware
• many options to represent signatures
• thought process: signature for Vienna?

  ---

• goals: compact, fast to check, reliable

aside: signature types

• one goal: detect malware without it running
  • examine code+data
• our first topic

  ---

• alternate idea: detect running malware
  • examine operations performed by software
• we’ll revisit later

exercise: signatures for Vienna

jmp 0x0700 /* C */
mov $0x9e4e, %si /* A */
...              /* A */
/* more app code */ 
...              /* A */
push %cx
mov $0x8f9, %si /* C */
...
mov $0x0100, %di
mov $3, %cx
rep movsb
...

...
add $0x2f9, %cx
mov %si, %di
sub $0x1f7, %di
mov %cx, (%di)
...
mov $0x288, %cx
mov $0x40 %ah
mov $si, $dx
sub $0x1f9, %dx
int 0x21
...

pop %cx
xor %ax, %ax
xor %bx, %bx
xor %dx, %dx
mov $0x0100, %di
push %di
xor %di, %di
ret
/* virus data */

/* C */ = constant changes when Vienna relocated
/* A */ = application code

simple signature (1)

• all the code Vienna copies
• … except changed mov to %si

  ---

• virus doesn’t change it to relocate
• includes infection code — definitely malicious

signature generality

• the Vienna virus was copied a bunch of times
• small changes, “payloads” added
  • print messages, do different malicious things, …
• this signature will not detect any variants
• can we do better?

simple signature (2)

• Vienna start code
  • weird jump at beginning??
• problem: maybe real applications do this?
• problem: easy to move jump

simple signature (3)

• Vienna infection code
  • scans directory, finds files
• likely to stay the same in variants?

simple signature (4)

• Vienna finish code
  • push + ret
• very unusual pattern
• probably(?) not in ‘‘real’’ programs
• real effort to change to something else?

simple signature problems

• how hard is it for virus author to break signature?

making things hard for the mouse

• don’t want trivial changes to break detection
• want to detect strategies
  • example: require changing relocation logic
  • … not just reordering instructions, adding nops
• need to detect signatures in real time
  • don’t want interrupt user (much)
• want to avoid false positive
• goals: compact, fast to check, reliable, general?

generic pattern example

• another possibility: detect writing near 0x100
• 0x100 was DOS program entry code
  • hopefully no legitimate program does this???
• problem: how to represent this?
  • describe machine code bytes
  • multiple possibilities

regular expressions

• one method of representing patterns like this:
  regular expressions (regexes)
• restricted language allows very fast implementations
  • especially when there’s a long list of patterns to look for
• upcoming homework assignment

regular expressions: implementations

• multiple implementations of regular expressions
• we will target: flex, a parser generator

simple patterns

• alphanumeric characters match themselves
• foo:
  • matches exactly foo only
  • does not match Foo
  • does not match foo␣
  • does not match foobar
• backslash might be needed for others
  • C\+\+ — matches exactly C++ only

metachars (1)

• special ways to match characters

  ---

• \n, \t, \x3C, … — work like in C
• [b-fi] — b or c or d or e or f or i
• [^b-fi] — any character but b or c or …
• . — any character except newline
• (.|\n) — any character

metachars (2)

• a* — zero or more as:
  • (empty string), a, aa, aaa, …
• a{3,5} — three to five as:
  • aaa, aaaa, aaaaa
• (abc){3,5} — three to five abcs: (‘‘grouping’’)
  • abcabcabc, abcabcabcabc, abcabcabcabcabc

metachars (3)

• ab|cd
  • ab, cd
• (ab|cd){2} — two ab-or-cds:
  • abab, abcd, cdab, cdcd
• \xAB — the byte 0xAB
• \x00 — the byte 0x00
  • flex is designed for text, handles binary fine
• \n — newline (and other C string escapes)

example regular expressions

• match words ending with ing:
  [a-zA-Z]*ing
• match C /* … */ comments:
  /\*([^*]|\*[^/])*\*/

flex

• flex is a regular expression matching tool
• intended for writing parsers
• generates C code
• parser function called yylex

flex example

        int num_bytes = 0, num_lines = 0;
        int num_foos = 0;
%%
foo     { 
          num_bytes += 3;
          num_foos += 1;
        }
.       { num_bytes += 1; }
\n      { num_lines += 1; num_bytes += 1; }
%%
int main(void) {
    yylex();
    printf("%d bytes, %d lines, %d foos\n",
           num_bytes, num_lines, num_foos);
}

flex example

        int num_bytes = 0, num_lines = 0;
        int num_foos = 0;
%%
foo     { 
          num_bytes += 3;
          num_foos += 1;
        }
.       { num_bytes += 1; }
\n      { num_lines += 1; num_bytes += 1; }
%%
int main(void) {
    yylex();
    printf("%d bytes, %d lines, %d foos\n",
           num_bytes, num_lines, num_foos);
}

flex example

        int num_bytes = 0, num_lines = 0;
        int num_foos = 0;
%%
foo     { 
          num_bytes += 3;
          num_foos += 1;
        }
.       { num_bytes += 1; }
\n      { num_lines += 1; num_bytes += 1; }
%%
int main(void) {
    yylex();
    printf("%d bytes, %d lines, %d foos\n",
           num_bytes, num_lines, num_foos);
}

flex example

        int num_bytes = 0, num_lines = 0;
        int num_foos = 0;
%%
foo     { 
          num_bytes += 3;
          num_foos += 1;
        }
.       { num_bytes += 1; }
\n      { num_lines += 1; num_bytes += 1; }
%%
int main(void) {
    yylex();
    printf("%d bytes, %d lines, %d foos\n",
           num_bytes, num_lines, num_foos);
}

flex example

        int num_bytes = 0, num_lines = 0;
        int num_foos = 0;
%%
foo     { 
          num_bytes += 3;
          num_foos += 1;
        }
.       { num_bytes += 1; }
\n      { num_lines += 1; num_bytes += 1; }
%%
int main(void) {
    yylex();
    printf("%d bytes, %d lines, %d foos\n",
           num_bytes, num_lines, num_foos);
}

flex: matched text

[aA][a-z]* {
               printf("found a-word '%s'\n", yytext);
           }
(.|\n)     {} /* default rule: would output text */

flex: definitions

A        [aA]
LOWERS   [a-z]
ANY      (.|\n)
%%
{A}{LOWERS}* {
                printf("found a-word '%s'\n",
                       yytext);
             }
{ANY}        {} /* default rule would
                   output text */
%%
int main(void) {
    yylex();
}

flex: definitions

A        [aA]
LOWERS   [a-z]
ANY      (.|\n)
%%
{A}{LOWERS}* {
                printf("found a-word '%s'\n",
                       yytext);
             }
{ANY}        {} /* default rule would
                   output text */
%%
int main(void) {
    yylex();
}

flex: state machines

%%
foo     {...}
.       {...}
\n      {...}

state machine matching

why this?

• one pass matching (except for some backtracking)
  • can make state machine bigger to avoid some backtracking
• basically speed of file I/O
• handles multiple patterns well
• flexible for ‘‘special cases’’

  ---

• real anti-virus: probably custom pattern ‘‘engine’’

Vienna patterns (1)

• simple Vienna patterns:

/* bytes of fixed part of Vienna sample */
\xFC\x89\xD6\x83\xC6\x81\xc7\x00\x01\x83(etc.) {
        printf("found Vienna code\n");
    }

Vienna patterns (2)

• simple Vienna patterns:

/* Vienna sample with wildcards for
   changing bytes: */
/* push %CX; mov ???, %dx; cld; ... */
\x51\xBA(.|\n)(.|\n)\xFC\x89(etc.) {
        printf("found Vienna code w/placeholder\n");
    }
/* mov $0x100, %di; push %di; xor %di, %di; ret */
\xBF\x00\x01\x57\x31\xFF\xC3 {
        printf("found Vienna return code\\n");
    }

regular expressions are flexible

• for Vienna: lots of flex features we didn’t need

  • things being repeated variable number of times
  • one of list of possible characters (bytes)
  • …

• but viruses try to make pattern matching hard
• good to think about what we can easily match

making scanners efficient

• lots of viruses!
  • huge number of states, tables
  • copies of every piece of malware pretty large
• reading files is slow!

making scanners efficient

• lots of viruses!
  • huge number of states, tables
  • copies of every piece of malware pretty large
• reading files is slow!

handling volume

• storing signature strings is non-trivial
• tens of thousands of states???

  ---

• observation: fixed strings dominate

scanning for fixed strings

making scanners efficient

• lots of viruses!
  • huge number of states, tables
  • copies of every piece of malware pretty large
• reading files is slow!

the I/O problem

• scanning still requires reading the whole file
• can we do better?

selective scanning

• check entry point and end only

  • a lot less I/O, maybe

• check known offsets from entry point

• heuristic: is entry point close to end of file?

example ClamAV signatures (1)

• hashes

4b3858c8b35e964a5eb0e291ff69ced6:78454:Xls.Exploit.Agent-4323916-1:73
7873be8fc5e052caa70fdb8f76205892:293376:Win.Trojan.Sality-93158:73
f358d77926045cba19131717a7b15dec:293376:Win.Trojan.Sality-93159:73
48d4c5294357e664bac1a07fce82ea22:450024:Win.Trojan.Sality-93160:73
e4b8442638b3948ab0291447affa6790:293376:Win.Trojan.Sality-93161:73
df36dc207b689a73ab9cf45a06fb71b0:232448:Win.Trojan.Sality-93162:73
baaeeabc7f4be3199af3d82d10c6b39f:293376:Win.Trojan.Sality-93163:73
...

example ClamAV signatures (2)

• simple regular expressions (with hex, different syntax than flex)…

Win.Trojan.Vienna-1:0:*:5051e8??00{1-255}5b83eb??fc8d37bf0001b90300f3a48bf3558bec83ec7cb430cd21
Win.Trojan.Vienna-2:0:*:be000356c3*50be????8bd6fcb90500bf0001f3a48bfab430cd21
Win.Trojan.Vienna-3:0:*:50ba????8bf283c60090bf0001b90300fcf3a48bfab430cd213c02
Win.Trojan.Vienna-4:0:*:b440b900048bd681eac102cd21721f3d
Win.Trojan.Vienna-5:0:*:b904048bd681ea130352515350b4
...
Win.Trojan.Vienna-129:0:*:51b89b03cd213d01017503e9????ba6d03fc8bf283c60a90b90300bf0001f3a4

example ClamAV signatures (3)

• ‘logical’ signatures: mutliple regexes together:

Andr.Trojan.Pjapps-58;Engine:51-255,
    Container:CL_TYPE_ZIP,Target:0; 
    (6&0&1&(2|3)&(4|5));  // expected patterns of below
    3a39303333; // pattern 0
    696d6569;   // pattern 1
    616e64726f69642e6c6f67; // pattern 2
    77696e646f772e6c6f67;   // pattern 3
    4e6f6b69614e373631302d31; // pattern 4
    336c676f6167646d66656a656b67666f733974313563686f6a6d; // pattern 5
    0:646578 // pattern 6: "0:" means must be found at beginning of file

playing cat

• harder to fool ways of detecting malware?
• goal: small changes to malware preserve detection
• ideal:
  • detect new malware
  • detect things malware needs to do accomplish their goals

detecting new malware

• look for anomalies
  • patterns of code that real executables ‘‘won’t’’ have
• identify bad behavior

detecting new malware

• look for anomalies
  • patterns of code that real executables ‘‘won’t’’ have
• identify bad behavior

viruses and executable formats

defeating entry point checking

• insert jump in normal code section, set as entry-point
  • potential counter-heuristic: run in VM, see if switches section
• add code to first section instead
  • perhaps insert new section at beginning

heuristics: library calls

• dynamic linking — functions called by name
• how do viruses add to dynamic linking tables?
  • often don’t! — instead dynamically look-up functions
  • if do — could mess that up/lots of code
• possible heuristic: look for API function name strings

evading library call checking

• if anti-virus software looking for function name strings:

  ---

• modify dynamic linking tables
  • probably tricky to add new entry
• reimplement library call manually
  • Windows system calls not well documented, change
• hide names

hiding library call names

• common approach: store hash of name
• runtime: read library, scan list of functions for name

  ---

• bonus: makes analysis harder

detecting new malware

• look for anomalies
  • patterns of code that real executables ‘‘won’t’’ have
• identify bad behavior

behavior-based detection

• things malware does that other programs don’t?
• modify system files
• modifying existing executables
• open network connections to lots of random places
• etc.

  ---

• idea: monitor for “bad” behavior
  • maybe run in VM and look for bad behavior

hooking

• hooking — getting a ‘hook’ to run on (OS) operations
  • examples: creating new files, modifying executables
• ideal mechanism: OS support
• less ideal mechanism: change library loading
• less ideal mechanism: replace OS handlers
• less ideal mechanism: debugger support

hooking

• hooking — getting a ‘hook’ to run on (OS) operations
  • examples: creating new files, modifying executables
• ideal mechanism: OS support
• less ideal mechanism: change library loading
• less ideal mechanism: replace OS handlers
• less ideal mechanism: debugger support

Linux hooking

• several possible mechanisms
• tracepoints, kprobes
  • cause hooking functions to run when kernel functions called or return
  • hooker function can arrange for logging or other action
• seccomp BPF
  • allow hooker to write ‘program’ to examine system calls of selected processes
  • can deny/change/log those system calls

aside Linux eBPF

• eBPF = extended Berkeley Packet Filters
• little programming language originally intended for network filtering

hooking

• hooking — getting a ‘hook’ to run on (OS) operations
  • examples: creating new files, modifying executables
• ideal mechanism: OS support
• less ideal mechanism: change library loading
• less ideal mechanism: replace OS handlers
• less ideal mechanism: debugger support

changing library loading

• e.g. install new library — or edit loader, but …

  ---

• not everything uses library functions
• what if your wrapper doesn’t work exactly the same?

hooking

• hooking — getting a ‘hook’ to run on (OS) operations
  • examples: creating new files, modifying executables
• ideal mechanism: OS support
• less ideal mechanism: change library loading
• less ideal mechanism: replace OS handlers
• less ideal mechanism: debugger support

changing exception call handlers (1)

• data structure tells hardware where program requests go
  • typically array with one entry per request type
  • usually set to point to OS routines
• simpliest mechanism: edit that data structure
  • and save a copy of what was there before
• point to your “hook” code
  • and call what was there before after behavior check

combining heuristics?

• one idea: look for a lot of factors about behavior/etc.
• combine to get “is malware” detector