hashlookup format
Computer Incident Response Center Luxembourg16, bd d'Avranches
Luxembourg
L-1160
Luxembourg
+352 247 88444
alexandre.dulaunoy@circl.lu
Computer Incident Response Center Luxembourg16, bd d'Avranches
Luxembourg
L-1160
Luxembourg
+352 247 88444
jean-louis.huynen@circl.lu
Security
This document describes the hashlookup output format used to express meta information of hash values seen in databases of known files.
The output description includes a common semantic. The hashlookup format is used by public and private digital forensics investigations services.
Introduction
Digital forensics is a critical field in information security and especially incident response. Providing intelligence
about known set of files is crucial to avoid wasting efforts while conducting digital investigations. hashlookup format provides a common output format for
diverse known databases of file hashes. Those databases are, for example, the NIST National Software Reference Library (NSRL) or
Known File Filter (KFF) lists used in digital forensics software.
Conventions and Terminology
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 .
Format
Overview
The hashlookup format follows the JSON format. The intent of this output to be easily
parsed by machines or generated by software in stream mode. Each JSON object is expressed on a single
line to be processed by the client line-by-line. Examples of JSON output are presented below.
Fields format
The main goal of the hashlookup format is to share common fields and to easily combine results from different sources.
As there is a wide variety of sources with various granularities of information available, the hashlookup format has been made quite lax
regarding the mandatory fields. The only condition is to have at least one cryptographic hash or fuzzy hashing value MUST be present in an hashlookup JSON object.
Cryptographic hashing
The cryptographic hashing value MUST be a JSON string. The string represents the hashed value of the file represented. The string MUST be the hexadecimal representation of the hash in upper case.
Fuzzy hashing (Context Triggered Piecewise Hashing)
The fuzzy hashing value MUST be a JSON string. The string represents the hashed value of the file represented.
Additional fields
Additional fields MAY be present to describe additional metadata from the file. The value MUST be a JSON string.
- FileName: Filename of the hashed file (NSRL)
- FileSize: FileSize of the hashed file (NSRL)
- CRC: CRC of the hashed file (NSRL)
- SpecialCode: Special code of the hashed file (NSRL)
- OpSystemCode: OpSystemCode of the hashed file (NSRL)
- ProductCode: ProductCode of the hashed file (NSRL)
- PackageName: Package Name of the hashed file as seen in metadata of Debian pakage format, RPM or similar package managers (CIRCL)
- PackageMaintainer: Package maintainer of the hashed file as seen in metadata of the Debian package format, RPM or similar package managers (CIRCL)
- PackageSection: Package section of the hashed file as seen in the metadata of the Debian package format, RPM or similar package managers (CIRCL)
- PackageVersion: Package version of the hashed file as seen in the metadata of the Debian package format, RPM or similar package managers (CIRCL)
- KnownMalicious: List of source considering the hashed file as being malicious (CIRCL)
Relationships fields
Two OPTIONAL fields parents and children MAY be present to represent the relationships with other hashlookup objects. The parent or children field MUST be a JSON array.
The value is a JSON string representing one the hashing algorithms. The SHA-1 is the RECOMMENDED algorithm for the relationship. Other algorithms MAY be used if SHA-1 is not available.
Sample hashlookup output
Binary file
{
"CRC32": "B4DD44A4",
"FileName": "./bin/ls",
"FileSize": "110080",
"MD5": "945FEDB3A3C290D69F075F997E5320FF",
"OpSystemCode": {
"MfgCode": "1006",
"OpSystemCode": "362",
"OpSystemName": "TBD",
"OpSystemVersion": "none"
},
"ProductCode": {
"ApplicationType": "Operating System",
"Language": "English",
"MfgCode": "534",
"OpSystemCode": "599",
"ProductCode": "163568",
"ProductName": "Vinux ",
"ProductVersion": "5.1"
},
"SHA-1": "5848386F77B4C60319C68B69C4594E29959381A2",
"SHA-256": "08AC13B08BFE4407E0F0C2E12E7F5B1B5E77EB817349A5EA1D836E83CD5ACB13",
"SpecialCode": "",
"parents": [
{
"FileSize": "1090622",
"MD5": "10A2318BE86F38A6ED113E16AABAA76B",
"PackageDescription": "GNU core utilities\n This package contains the basic file, shell and text manipulation\n utilities which are expected to exist on every operating system.\n .\n Specifically, this package includes:\n arch base64 basename cat chcon chgrp chmod chown chroot cksum comm cp\n csplit cut date dd df dir dircolors dirname du echo env expand expr\n factor false flock fmt fold groups head hostid id install join link ln\n logname ls md5sum mkdir mkfifo mknod mktemp mv nice nl nohup nproc od\n paste pathchk pinky pr printenv printf ptx pwd readlink rm rmdir runcon\n sha*sum seq shred sleep sort split stat stty sum sync tac tail tee test\n timeout touch tr true truncate tsort tty uname unexpand uniq unlink\n users vdir wc who whoami yes",
"PackageMaintainer": "Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>",
"PackageName": "coreutils",
"PackageSection": "utils",
"PackageVersion": "8.21-1ubuntu5.4",
"SHA-1": "F335B669CCB7BA8A2FC8FAF315B1B4BFF9D4217F",
"SHA-256": "07995A739DAEBD60297F0E9C2B44DFAB0C735A0FE08FACC097ECE06BB4B9FA0B"
}
]
}
Binary file - package
{"FileSize": "1090622", "MD5": "10A2318BE86F38A6ED113E16AABAA76B", "PackageDescription": "GNU core utilities\n This package contains the basic file, shell and text manipulation\n utilities which are expected to exist on every operating system.\n .\n Specifically, this package includes:\n arch base64 basename cat chcon chgrp chmod chown chroot cksum comm cp\n csplit cut date dd df dir dircolors dirname du echo env expand expr\n factor false flock fmt fold groups head hostid id install join link ln\n logname ls md5sum mkdir mkfifo mknod mktemp mv nice nl nohup nproc od\n paste pathchk pinky pr printenv printf ptx pwd readlink rm rmdir runcon\n sha*sum seq shred sleep sort split stat stty sum sync tac tail tee test\n timeout touch tr true truncate tsort tty uname unexpand uniq unlink\n users vdir wc who whoami yes", "PackageMaintainer": "Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>", "PackageName": "coreutils", "PackageSection": "utils", "PackageVersion": "8.21-1ubuntu5.4", "SHA-1": "F335B669CCB7BA8A2FC8FAF315B1B4BFF9D4217F", "SHA-256": "07995A739DAEBD60297F0E9C2B44DFAB0C735A0FE08FACC097ECE06BB4B9FA0B", "children": ["9799864E326E9DB68121471C6E72EA45152BD2C8", "A880A1F35311A7D34C9B2CA10418BD6EE94EFF58", "3E9FE88BBFE594A701522C2BAF65E209FDF7EFD9", "E03605C7911BF75BE61E54FB922B11DCF1EAAFF9", "181A05F9D249BC99FB684984C631F149CC96990C", "7EF60EF3D83B352ACC9AF996ACDC7BE3DE955705", "C687BCF2FCB74F649163AAB837C15D5800D80B31", "168CA019316332AC0A01472BDF8769801F613DCA", "6645E63EC4411B54287CB0C1321160E44B05FB87", "1989E263AF6ACF6E2869D0B4CF9433E240213C4E", "DEDFD0DA98EA7D07A5B1C7D16EACDCD0154C79B9", "C1A0868024358B0C2F0A2991BD4676B70EBF66F3", "69952AAC37372161F66DA9B07FE0BE5263A9249F", "C03D2998DE9E4332DC91586F00DAC3CA5F4761C3", "E558E7B478FAC50CCACE0E6BE28CAA19FBC74D4E", "08E3AFA387417CB0DA343837D68374A4AB8D915F", "46750D0726DB8D8E4DF870B1060E2414B64164E9", "47A2C7B8518F7E790B097785E25C8E5909AE8A39", "9D8E3FD5BE0D04815F5B0606E94C3B7C29CC8788", "FCAD50919AF0544353E950D1B52E6E2C7B98854B", "CFDE9C955B1E774EF72FB2B4A3E180EADF4CD3D0", "95C37C6826FF0C4E1C17D08285D346EE643343C0", "B35FB1B6DCEC924603F8A86677DBB0D54F5B3C18", "F8592BBAF848DEF6DC26266B15246C50D45C2095", "FF43C49690FF5E5D05CA59D7E66CEEA7E3CE00D0", "903FA7065E10EA3ED8D07598715E7AA73C38BA5A", "64D49DB459A03800399A5BB8CFFE979A604959CF", "688FCF82E39A1C44424FE9F440B69B341FC8B6E2", "0A073CD0BBFB5A33F539003F85E4D90C4CA3F2BD", "C158DC6563C36EA34408E1A36D9786A8CD95FD22", "6ECEBFE6C408316371834A52A037D91EB5BD2A1E", "2411EE623576B90A80AF6B50C47E6186F8AB6308", "2592E88C4FD1BD34207E480AEF99508090370945", "3904F363902225998E2222D67D20D01579461C15", "76FCB8813682CC8697AF1E5C6DDD5FB1DFDEA23C", "D461D21F9994EA40C68651BCA6A6D07F43A551C4", "AEE56A85B66D037EAD8C2D630194C8C46E5C061A", "69AD64E3E922A40EC3372C5DBEE57E8ABE486227", "1348C730C44DF01C0D49DB6084B5736A1D7A3BBC", "E0A97820A852729E236F7524CC23060C7BD7638E", "BF2AAAEDAB78605C43FF12450381C07C15A01D06", "071E525B285AD74B3884F1661D857E8491D2C622", "087204A60FDF211A545A5B8B900F23C0AA118333", "5848386F77B4C60319C68B69C4594E29959381A2", "E19EC3D5E7B9AEC5320CE69FAE5A61EAA7AB2143", "A57FB1F8263E8AA32B36B0D08CDA214E55DE9202", "E8FC405D941E589AA1BAB01E05AD321A049B7775", "A9DB474B1F26FAF5E01B7D545D9CA66269487053", "5E49F9779E165B4EEE965914FECBB8F7D78C59C8", "B90EAF06CF7C6F829B5D671936B59052797F78AC", "90772F507D44570ECDF615B4C232F19FC6DD9D0D", "8025FBA333B4ADCE8EFCAD284541F38C41DA09DE", "F899AA531C826504B9B494C8EA5E4532313210A0", "6AE02146CEC01123843393817D352A805227807D", "3F9E530FB7E8C37E53FFCB1ECF977E00C25F224E", "416FD291494F58795077E5EE064D92329E15DE0E", "9A9A76DA8EC26B5AA3B1F35C434466291424718F", "731D8E470227AC5AAD8432EC123984B9D052B7A0", "EBE777F05B387155604065AD4BED08D2EA3CB8B8", "787ACF5E6A560B4AEB008111F701A730AADF2EC7", "AA6F74DAD038A0DFCD4D64A002482F40BB732F10", "16C68C5BD71D93E06C2D8FC7F0F0F73954C6D7C3", "4AA2F5D8C293531D72137EAE6F71044CD46B5AD7", "99F256008B4847D716492E9763D03D049EEECE6C", "3244B00D11BA9997243A3C2D3108FC915A4BA042", "9C7C38A4B21AACB6DC51AB8A97F6CDE6704ACCB6", "781A6E4FE0CB8167CE423FC476240BDED698D676", "A71230EDAE1E5D354C672E7AB1CE92BC6059EC7C", "ACBD8B51B76889C2F55820E1C32BB26FBF67C441", "7C394AF4519CE500161DEB17857B9C057B7C74B3", "7AB1711D45DB3752B7CBE446D2E0E62D77E75EE3", "8CD8BD2875A68CD5E01F4A071B3F39F5B725B2B6", "D5FF3DB00A37075C07878A718852AFE539C7610D", "1B56B7A2684EB25235DE1708ED1371CD7879540F", "53D8A8EF35DA82BB6118BF9D8BF1ED404FC383C9", "47392375E355F56961CA42E1CFB6B98BBD484D4C", "DCBD08101D550F76DCC1B7507B152734B6F7CE7C", "F8DF08929A667BED6D7086812C319DB522E8292C", "1FB274F0883E0075D398806ACD0FA765F1118C20", "583A638A220E6FD91569F4E263771D4F89F2CC67", "39698F9FA30AE6990ACDF69280B682D90705EC32", "DE762C72E9720DB70C1897C3E0FC9C8C7D160210", "283DD2D206477E4081911F6B83950EE76503EBAD", "A7C8E7F93AE9E86A7836238644274CD73C75F5A7", "A03537A232503853D9D4C30C732CFB5F12913D93", "3F4D980B4870D5A6EB3DF334CA49C5566000C97F", "C028089A484BA0AD0166281B58DEBE0C99E5B4E7", "4A46ECF0336D55076B1C72D2459139F85DF7236C", "DF9C0BDE30B1E2C8FAF8E35CBBD3672C0AE0DFDA", "29E52E423E17C0CE93D38AA8380B0A3AF137BC7E", "BA5FD03FF246DE181DDCAA744DC2AEF4D3711762", "FE7D945A453A276E1CD005AB7A1E177D48C63A86", "DD5A833CD33617F4DCDA0220809D41FA9B1EBFC7", "4017F5B69FBB5064E51A95C856437859BA6CBA83", "427B32F83A3FC5CA0813CBACE975ECA2D6AD918B", "8F16C526AF56EF4369611718463AD4975811DD20", "FC1C867CC2D89C5985C8CB833A3980915AF7E1A1", "EB3FCFD28830CCED7C6BAA04908F574EA4F2A61F", "410D1E55EE08EF6BD7DE39DE80A02981BAF151D1", "988B560C670EDD9E2AEEE6C1D6DE584518CC57F5", "F26ABF496FC9940BB06CD4CEC3D892CF2E426B6A", "22AC7B8CA7324408A18BB95FB55AAE9E5EA85785", "F3E09D476BB01366D740EFF92453AC73C8356F43", "6DD5ACA0C43EF39FD7FFAFEEDE46986985BC84DE", "E38EDB72D805F466556C8BB796EA729CBCC04245", "D6C447B56B702869E9B429A7B47E1CE3B57BED9A", "64E23452FA980EB5DD1D4375071CB6124CF9B196", "7883EB75715603F1B89449BE617F91C65698DF38", "C328540F2D947D50EF02AC958C0DCDB51CCA0DD0", "2AC91E34FE455B026B537EA8FBA86E69E251CEE4", "672D844C60553F9B3DB9844DC29DDF49BC426F45", "EDB7ED42160C95BA2C9DD4C1E87577BD85DFE5FB", "6F4DF90B509C9392A180D7C76BB0058D4C44A4BD", "1C75F8FAF710C17B87DFD75D8390F2F2F7AF6400", "C02435F5BE0DB85C12B47E33644322631CBE7CCA", "B0C07FAC1BA571EC3054D79C40181D99CB4D01ED", "CA0B3EBCD75D8799863ADDB66E9BD378A3B88F8E", "521E4CC97D2372F821F75DB03A26938F923C002C", "CFE1C6F349F1F0611CFD3B6DD0E60EB135D7D798", "DFC1F4C951F6E09B6CFCA69127BE483279A9B5F9", "6E957B4561B081DA16886751D729170975D860BE", "A570E581D7E1D5308E88154967C3BDE3593DA50D", "99CCB36AE5BA1B6EF528C3477CDB1185744C8DA7", "A07770FC93BAA888407523BB58EE9CE97C94DB02", "56035A58036F19A2C9D312BF2730C7F970B380CA", "2C3D857765A05AF072146796B07D6A063C0B7224", "4BD3020D460E50386297CE14F9AD85B7278F9A73", "7F8BF064B328BE934582341857413CE7A387CB81", "403645FFD3A5B16F3E558947C1854FAC180C1E92", "CC125B501A779D230063BA38A7BDEB93041D82D3", "9FC7AE958DAB28FE8B6466555BDFF954BE0EC2E7", "BAE0BFB8B27FC8806194C299435AD578BC93731A", "D357E7C22254E182377A57350BB9EC870B677B32", "9C3290446A139A29D000D920E83AE5ABA264C89A", "D3582A8EF775B8E3FB4771B7B6762FBF388C131E", "DBC34938767985B8C06471483D794A1ED91529A8", "23BEFDFDBD4FDE0052EE71D6E5561F2756C85F91", "53360CB53781CACC39BED7A4484A9B8AF7D356F4", "3E98B06F33CBB14590231B74BBC277418605BE21", "C41D8DEADD83B2FFDE06CD517452A680A87A44F3", "5942742E2461BF2646FDFA48C44F1BFDE7EC37EB", "D74E21CA765F9B05DE6535799A68DCA14DE3036B", "70090B7400AC6E18BE5E1C05FB6C0EC19EBB2B5A", "AF5BE22C07A95E8D7464415A5B988D7B46F34018", "24E177F4DDB835CBA8A12F9E1372E2338A2891E9", "D805D555362790B465829C15296CC9382898FAAC", "543D2D3AA93FAC487EAD738460539FB6EC9D8D52", "B20FEC67D2246000FD86FA211DD40CAAECCEB9AD", "7A806948FE658A5BAC29A0F5CA3710533D848565", "DC949A8C7FA9FEE929930B6FB599BC82F8DC4C5A", "55DBEAFBD6A4CA81E110CF0213051C11006AC4C9", "844D6D1DB5060B26976F07C66F2F8AAD2E455F65", "E7DAF9F24A6B790F157203235278CE3F4208CB45", "B32B2AC7FFE8F658378ECA63DCA037C64A867C95", "4326E056CE3813801B5DA2848248840D2E317C94", "CEC087401C965D8AF9D4DCFBDED5AD305C86697D", "EDF9B4EABA52E2A5570D936AD74C142E3CCE1CE8", "C04C8D4891153FAF0CF8E27CBC31A556862AB783", "47073CD75ED4721EEB6FF29C52D5D871771C57A4", "BEC391C559DAE7FFC0B5D90EA7CA65028AA1D16E", "5A11AC51F28545D7965990027CA63F28C410A51A", "66F6BE00855E7A2CB618F52C99F795A055534911", "75C44CAD971780BDCE8CE499F7ED7CA235985EC5", "87AA6E1E92AA2552DC6E431E88C22774E565F14E", "21C0A42179BF4A6C2E58DDF1A1BF58C668830A50", "8AF3BAAD4074A5267E6E7D8D77D0F0B0AADFEFC4", "5B74838880C11A9FEF94FDA3964DD6BA53F812E7", "F5C5CF75A119FF8818283B9CC932217888CDC8A9", "1056EBBFBBBF2E51BCB8A48FF8038CF66095F63A", "55016AD7A449B91A5DBE59308BDD3E84261A8FF9", "F0BC5D070A82BBCD1749EBE219671FEEEAEDAED6", "3884371FFF88B50CB05D50549A1A3C106017862C", "5EEC0C5A87A28238955970C668BA4DD733A27380", "56A361D047DB9C25736638CA6C8DE5D8F5AC4A1D", "7AD82880980897DE4B9E08DFC62E7E378426F91B", "3F3AACED5AAD06D9591F9B4EDC8DA1D87CF608E4", "21E08482B1CD04B02D37FA6AC4D8B562E684E52A", "9E3B379E28C7C00AE175F23F63555EA2111D4447", "B0E6C075C30FE3F28649AF644B5BD54A4CBDC509", "B9BFA0260EA37824C42047C3EC80C83A2BEBFAC8", "DA1AE2BDBE1281B7D4E3E10D828FC370BD12572A", "3988BFCAA09F78083C23996B9D049269CF088CF2", "C2BEBF667833B55C5495753657C469AD07332007", "3E4FDE8DECFD58B5273B2B72A5D2D67804AAB27C", "3468D6009DA54AF9C6BF3E78D058D87886C9C6D2", "A6817340664ACE9688B4E9399A08024788AC03D7", "4FEC0C2432C2DF1EF03A8A63CE346179FCF3D1E4", "AB304FDC233C801E3D4129896F49A04D0F33C7E7", "B18D970EC65694033FD489FFE297ABAB9B391EC9", "8898C5F7E6CD1A1534080CB6D96003CBF0E0B5F7", "74D1EE107C5C63D4ADF5C1902F4EBD56EC1A7669", "084051009844302F09B1D5D48C2BA73F54BC8FB1", "250D7DA9FDED702FC1B3A127756367841CD851C6", "45B47C75BFE354E4DA4E6B101E4426634E56A927", "BA971742ACBB679EB80C20D0942C9F95D6BD6B7C", "861C814423F49B97077D4910FAB0C02D54EAB4B1", "44B78AF79C57BD5235883D15EC3106F96A2A5AA9", "BCF5A7EB0FD6362BB317BA69D7925002C7E1E3BE", "E0878F0243391A4537E0DF1652BE8D506FA749D1", "134A47F790EE082AB9A7E2503F01B0C164D777E8", "9352FF68CF7B5E73E7434BF138AFC2E17FB4545B", "3E6791CD56A48FABC6F637BE11D234E8068EF91F", "6F6A5B2A733ADBBB1100C44233DFBD3C5D00E4EA", "BB7E54487E24D3778C2C0EAB965AA421EA9D4D0F", "B7A599586D2BD1B69C69EB0862BC665DAAD9FC61", "2B129FA27E458F767898FD152EB65047B65238FE", "4D34641032551FBEBFFDAF751B707B90F9570C4D", "912A4B72F98C0EBEF5C235A55F49BA5EC5E463D9", "32D82E8E9873FEA2613B882461C58E13AB6BE52B", "F0C41EB583D4B17416DD70F1079E6FDEACE144FD", "014D78C0ECB1ED495D12E4FA0DA9C7A6A953945F", "4E492E947E446DF5D4B19AAB08664D65A3E604AE"]}
Implementation
A public hashlookup service is provided by CIRCL and accessible as a ReST HTTP API. A software back-end implementation which produces a hashlookup format
output is available .
Security Considerations
hashlookup results events might contain sensitive or confidential information. Adequate
access control and encrypted transport layer shall be implemented to ensure
the confidentiality of the hashlookup results.
hashlookup results don't imply a specific assumption concerning the maliciousness or non-maliciousness
of a file. hashlookup only provides the information about the presence of a file in a specific set, known source or database.
Acknowledgements
The authors wish to thank all the users of the CIRCL hashlookup services for their feedback.
Normative References
Informative References
hashlookup-server is a minimal and fast open source server (ReST/API) to lookup quickly hash value from large dataset.
CIRCL hash lookup is a public API to lookup hash values against known database of files.