/* Staged Block Mapping format to understand how the resource commands in the XHyperDBTools extension work IÕm not sure if Apple will ever support this -- before you go any further... DISCLAIMER Certain names below may be trademarks or registered trademarks of their respected owners. They are added in this format for suggestion only. Therefore, I am in no way related to their owners or the staffs of the companies that developed the software that use the names given. THIS TEXT CARRIES NO LICENSE, OR WARRANTY. I AM NOT RESPONSIBLE FOR ANY DAMAGES THIS FORMAT, THIS TEXT, OR RELATED SOFTWARE CREATES, WHATSOEVER. USE AT YOUR OWN RISK. 2004 Jan 13 Did the full conversion to Bare, and did more work on the format 1/15 Moved the application specific flags out of the map specific flags. Added an ÔextendedÕ flag, and took out the ÔhasFlavorsÕ flag, since it can be determined by the value of the ÔflavorListStartÕ field, and/or what it points to. Rearranged the header to make it more application friendly and filled in the rest of it to make it 128 bytes (two to the power of eight). Finished the first prototype of the resource entry. Made the extended resource block length 55 bit size (instead of the maximum 63 bits, to ensure space for other resources), and added an extended ÔlockDataÕ flag, to give the user an option to keep the data in place, for the possibility that a HUGE amount of data may be requested to move. Did some ÒcosmeticÓ changes to the documentation. Added some future API names. 1/16 Moved TypeInfo out of the typeBlock structure, and replaced itÕs designation with a reference to it. Did some more ÒcosmeticÓ changes to the documentation. 1/17 Wrote the f**kinÕ disclaimer, to prevent getting sued from the Suggested names in the ÒTypeInfoÓ structure. The name of the programming language could change due to the possibility that itÕs already been used (there are so damn many!!). I may add code that will make it possible to include Library files, so that certain XMap format handler UPPs are loaded at startup. Notation... As long as the first first four bytes are the signature 'XMap' then the file uses XDBTools to manage the file format. The format is customizable, so theyÕre ssSPECIAL. The HFS FileType is not required to be anything specific!!! Never, did the MacOS resource manager, have the ability to handle such a complex format. The tools extension handles error reports, so individual bits in the report structure get set if any thing abnormal happens. All errors (size dependent) get reported in the report structure if not NIL (in memory, of course). Remember, only a negative OSErr returned means that the file couldnÕt get opened. A positive value may be returned to signal other things. */ #import "XDB Tools" -- future command APIs - could change, so donÕt go changing your code prematurely extern OSErr InstallReaderUPP(OSType mapType, XMapHandlerUPP); extern OSErr InstallWritterUPP(OSType mapType, XMapHandlerUPP); extern OSErr InstallDecompressorUPP(OSType cmpType, XMapHandlerUPP); extern OSErr InstallCompressorUPP(OSType cmpType, XMapHandlerUPP); extern OSErr RemoveXMapHandler(XMapHandlerUPP); -- XFileRef is a 32 bit integer, by the way typedef long XFileRef; typedef long long XTypeSymbol; -- ErrReport pointers can be NIL extern xFileRef HOpenXResFile(VRefNum, DirID, Str255 FileName, SInt8 Permission, ErrReportPtr); -- if Ref is negative, then itÕs a VRefNum -- otherwise itÕs counted as a file refNum, and if ÔxFileRefÕ is also valid -- then the new XFile reference will be returned in the pointer extern OSErr CreateXResFileMap(short Ref, DirID, Str255 FileName, *xFileRef); extern void CloseXResFile(xFileRef); extern void UpdateXResFile(xFileRef, short flags); extern void FlushXResFile(xFileRef, short flags); extern void UseXResFile(xFileRef); -- uses the ÔinternalFileNameÕ field extern void UseXResFileByName(char*); /* CurXResFile get the current XResFile xFile reference and/or the ÔinternalFileNameÕ ÔnameÕ can be nil make sure that the pointer can store at least 64 bytes, if not nil */ extern xFileRef CurXResFile(char *name); /* SetXResFileInternalName set the ÔinternalFileNameÕ field If the ÔdontCacheÕ map attribute is set, then the 64 bytes will be written to disk, regardless of the ÔwriteNowÕ parameter. */ extern OSErr SetXResFileInternalName(xFileRef, char*, Boolean writeNow); /* CountXResTypes Returns a count of the given type, or type info, which either or both parameters can be passed. If both are zero or nil, -1 will be returned, and ResErr will be set to paramErr (-50). */ extern long CountXResTypes(*XTypeSymbol, TypeInfo); /* GetXResource obtain the resource item by generating a handle The attributes are accessable from the handle returned. The handle contains a copy of the item data only if ÔResLoadÕ is not zero. */ extern Handle GetXResource(*XTypeSymbol, TypeInfo, long ID); /* AddXResource add a handle to the resource item chain If ÔdontCacheÕ is set, the handle contents get written to disk. Otherwise, the handle is added to the map cache, where the contents get written to disk on update or when memory gets full. */ void AddXResource(univ handle, short attrs); /* MoveXResToStage return the index of the resourceÕs stage block it should never return a negative value */ extern long GetXResStage(Handle); /* MoveXResToStage Makes an attempt to move the item block data to a different stage. If a negative number is passed in the ÔindexÕ parameter, it would be counted as the stage ID, instead. The xNoChange bit is returned in ResErr if the item is already there. */ extern void MoveXResToStage(Handle, long index); enum { -- error bits returned in ResErr as flags when ResErr is positive xCheckFieldErr = 0, -- one or a number of the ÔcheckÕ fields are bad xHalfBooleanErr = 1, -- not all bits are fully set or cleared xInvalidModDate = 2, -- one or a number of the fields are out of bounds xNoChange = 3, -- not really an error; just, nothing happened -- let the user know, anyway }; --#pretype signature SBMDataType; (not needed) head struct XHyperDBMap(72) OSType = 'XMap', -- MUST have this constant value SBMHeaderDataType mapType.OSType, internalFileName.cs[64], -- on file systems with 255+ character file -- names, heh, this would be a Short name SBMHeaderDataType header.b[], data.b[]; /* The rest of this file describes the default ÔbinaryÕ format IÕve developed. It has the SBMDataType signature '\0SBM' -- or 0x0053424D in hex The initial file size (EOF of empty list) is always 128 bytes. Resources (or if you prefer ÒItemsÓ or ÒChunksÓ) consist of 64 bit type symbols and 32 bit IDs (like the Dynamic File Format), but designed to give much more performance than the MacBinary format. The list of types can be stored in multiple locations, rather than one, in whatÕs called a ÒtreeÓ. The type listing tag must be "DivsTree" in order for the type listing to act as a tree. If you prefer other listing types, there are less efficient single formats, at the bottom of the source header. Type listing blocks in the ÒtreeÓ format are scattered around in the file, usually adjacent to resource block data in the order of: typeList, resourceBlock, [IDList,] [NameList,] typeList,...etc. and continues until the ÔnextBlockPtrÕ is invalid. If there are new types in memory, then when the map gets updated, the new types get appended to already existing listing blocks at the most convenient point in the file once the file is updated. Resources/Items: In the standard format, the limit to the size of the resource data is the same as the maximum MacOS block size, which is 16 bytes less than 2 Gigabytes. But in the extended format, itÕs limited to one byte less than 32,768 Terabytes! Such a size is impractical, since on most file systems, it would go over the maximum length of the whole file fork! Flags: Every defined flag is expressed in 3 bit flags for protection. The ÔcheckÕ fields are for alignment and to determine if thereÕs any possible corruption, because they are in ÔfullÕ boolean format, so all bits MUST be zero or one, or else they will be counted as invalid. Other parts of the file format have their own specific designated value. Checking to see if those specific values or the other flags in the header are fully set or cleared can also determine possible data corruption. Encryption can be application defined. The default encryption format uses a random encryption format, which will not be explained in this document, for protection purposes (in the case that anyone will ever use this). Do not set this flag on disk, unless the rest of the file data is encrypted appropriately. Do not modify ANY data on disk when the encryption flag is set, or else data will be unmodifiable AND unreadable. The ÔpackedIDsÕ flag cannot be set while ÔhasTypeIDsÕ flag isnÕt, or else an error is reported. -- that wouldnÕt make sense, would it. If the ÔprotectedÕ map attribute is set when the file is in use, then a copy of new data will also be stored in the flavor data, and the owning type symbol will be stored before the corresponding ID list. ÒFlushingÓ a file with the protected attribute set would cause ALL item data to get duplicated, so be careful, Õcause the file could GET BLOATED. If the ÔhasCmpRoutinesÕ attribute is set, then XDB Tools will autmatically scan for compression and decompression resources that have the type info value of the matching routine type. The scanned results may be stored in the flavor data, for performance. You must set the ÔchangedÕ resource attribute, if you have changed the type info value on disk. Stage Count: Each block of resources/items are divided into stages. Stages are meant to allow some sense in to where each item belong. Stages can be used for a lot of different purposes, like in Gaming, they can represent levels. Individual item blocks can be moved to different stages using APIs. Checksum: Every time the map gets updated by an application different from the previous (does any valid change to the file), the checksum gets moved to ÒparentÓ checksum field, and a new random value fills the current checksum. Software Sig. (swSignature): Stores the creator type of the program that previously modified the file. This is necessary in order to update the checksum at the right time. */ struct packedDate(6) -- limited to March 5592405 A.D.!! monthAD:s26, dayOfMonth:5, -- zero is first secondsOfDay:17; struct extPackedDate(8) monthAD:s26, dayOfMonth:5, -- again, zero is first milisecondsOfDay:27, randomSignature:6; SBMHeaderDataType signature '\0SBM' struct SBMBinaryHeader mapFlags.l = { -- MUST use the appropriate APIs to change map flags! check:2 = 1, half encrypted:3, -- if set, -everything beyond this point half changesMade:3, -- update map? half extended:3, -- larger blocks, more flags half hasTypeIDs:3, -- otherwise, there are NO individual IDs half packedIDs:3, -- if set, all are packed in bitfields by the -- difference from the previous half hasNames:3, -- otherwise, no one knows what they might be half hasSymbols:3, -- otherwise, the ÔdataTypeÕ field is the -- only way to tell what it is- half hasTypeInfo:3, -- ... unless this is set half reserved:3[2] -- future use, set to zero } stageCount.l, -- could also act as Game-Levels mapAttributes.l = { check:2 = 1, -- can be modified at any time - use Sync calls half readOnly:3, -- locks out user from making modifications -- these change how the map acts during updates half compact:3, -- actively makes multiple copies of new data half clearUnused:3, -- actively clears unused data to make things -- more visible - decreases performance -- these add flavor data during updates half protected:3, -- makes multiple copies of new data on half recordChanges:3, -- adds (packed) flavors that contain info. -- on the changes from the last update half dontCache:3, -- all changes get written immediately -- the map is re-read on open half hasCmpRoutines:3, -- scan for compression/decomp. resources half reserved:3, -- future use, set to zero half appFlags:6, -- application use }, swSignature.OSType, -- the previous app. that modified this file modDate.extPackedDate, -- my dating format - contains a random number parentCheckSum:48, -- if zero, then this is a Fresh file! checkSum:48, -- gets set to a random number every time a -- different application updates it fileType.l, -- application defined offset flavorListStart.q, -- in some programs, itÕs called a ÒDirectoryÓ -- the type list starting tag - must be recognizable or Else the rest of the -- data is unmodifiable AND unreadable SBMBinary TypeListingTag typeListStart.q; SBMBinary struct TypeInfo(4) list:1, dataPacking:4, dataType:4, dataSpecType.w; /* The data type fields arenÕt required to be any of these unless it deals with concepts native to XDB Tools. Most of them are added primarily for convenience, and reference. Zero for all 32 bits is default, which would mean Òn/aÓ */ SBMBinary struct TypeInfoOnDisk(4) full check:2 = 1, check:3 = 0, half list:3, -- would make it a list of dataPacking:4 = enum { undefined, aligned, variable, compressed, terminated..reserved..none = -1 }, dataType:4 = enum { binary, -- no specific data, no specific header text, alias, -- points to another file info, image, graphic, sound, music, multimedia, -- animated graphics and/or sound code, scrap, modifier, other..reserved }, dataSpecType.w = enum { undefined, switch(dataType) { case text: plain, markUp, -- formatted with Ò<>Ó tags, like HTML styleRun, -- 32 bit integer text length, -- justification, text, style data justified, -- array of: 32 bit text length, starting -- point, ending point, justification styledJusText, -- same as previous, but contains a style -- tag format in between entries -- length, rectangle, text justification, 24 bit colors textBoxArray, styledTextBoxArray..custom, break case alias: MacOSAlias, -- OSType, length, data generalFSSpec..custom, break case info: comment, image, printer..other, break case image: generalBitMap, -- width, height and data boundedBitMap, -- rectangle(s) and data maskedBitMap, -- rectangle(s), mask and data generalPixMap, -- width, height, bit depth and data sourcedPixMap, -- point, dimensions, depth, mode, data boundedPixMap, -- rectangle(s), image info and data maskedPixelMap, -- bounding rect, bounded/srced pixMaps tUniversal, -- my format! TIFF..custom, break case graphic rectangle, polygon, color, QuickDrawPICT, Photoshop, JPEG..custom, break case sound: raw, -- 8 bit PCM, usually 22KHz sample rate CDAudio, -- 16 bit, 44.1KHz generalPCM, -- sample rate, data MacSoundHeader, MacSoundList, AIFF, WindowsWave, IMA..custom, break case multimedia: QuickTime, AVI, MPEG..custom, break case modifier: styleData..custom, -- points to other text break case code: raw, UPP, external, -- as in XTNDs, or HyperCard XCMDs processSegment, -- as in, CODE resources driver, patch, trap, -- system types /* Compression and decompression resources must use the ÔcompressorÕ or ÔdecompressorÕ types, and make sure that the map attributes flag is set in order for XDB Tools to automatically load compressed data with their corresponding code signature. Otherwise, they will have to be registered manually, at runtime. */ compressor, decompressor, other..reserved, break }, }; SBMBinary struct typeEntry full check:2 = 1, idListOffset:30, -- offset from the end of the type list block global hasSymbols ? typeSymbol.q, global hasTypeInfo ? TypeInfoOnDisk; SBMBinary struct typeListBlock count.l, entry.typeEntry[headingCount+1]; /* When the marker after the type listing block is zero, it follows an offset that points to the adjoining resource item block. The filler usually doesnÕt get created unless the map is not set to automatically ÔcompactÕ or the pointed resource block has itÕs ÔlockDataÕ set. */ SBMBinary struct typeListBlockFiller marker:1 = 0, itemDataOffset:31; SBMBinary TypeListingTag { signature 'DivsTree' struct typeTreeEntry typeListBlock, -- file offset to next type list block global extended ? nextBlockPtr.q : nextBlockPtr.l; -- single types (less efficient) signature 'TrSingle' struct typeListTrailing typeListBlock; /* typeListLeadingSingle is the least efficient, since the ALL of the data after it has to be moved in order to add new ones */ signature 'LdSingle' struct typeListLeading typeListBlock; -- new listing types may be added in future releases } SBMBinary struct CompressedResourceDataHeader alt(CmpRsrcDataHeader) SBMCmpDataType cmpType.OSType, length expandedLength; SBMBinary struct ResourceAttributes(2) reserved:7, -- future use - set to zero hidden:1, -- does not show up using GetIndXRes,...etc. disabled:1, -- disables data from being read from disk -- MacOS load attributes sysHeap:1, -- otherwise, application heap canPurge:1, -- ÒpurgeableÓ is not a word loadLocked:1, -- the handle is initially locked when loaded protected:1, -- cannot be written back to file -- in some cases, cannot be overwritten in memory preloaded:1, -- if ÔResLoadÕ set, then automatically load after -- the map is read needsUpdate:1, -- update attributes and/or data /* WARNING: ÔcompressedÕ resources, like in the MacBinary format, require the header to be correct. Do not set this attribute on disk unless youÕve correctly written both the header AND the compressed data or Else it will NOT load properly. */ compressed:1; -- can also be used to hide data on disk SBMBinary struct ResourceAttributesOnDisk(4) check:2 = 1, reserved:3, -- future use - set to zero hidden:3, disabled:3, sysHeap:3, canPurge:3, loadLocked:3, protected:3, preloaded:3, needsUpdate:3, compressed:3; SBMBinary struct ResourceEntry /* The marker is necessary in order to determine whether a resource block starts here, directly after a possibly existing type list. Otherwise, the space contains an offset to the starting point of the resource block. */ marker:1 = 1, -- brackets must be used in this case, since there are bitfield expressions global extended ? {full check:5 = 1, half lockData:3 = 0, size blockLength:55}:{size blockLength:31}, attributes.l = ResourceAttributesOnDisk, local compressed ? {CmpRsrcDataHeader, data.b[blockLen-sizeof(CmpRsrcDataHeader)]}:data.b[blockLen]; /* the ID list block comes directly after the block of resource items resource blocks are in corresponding order to their ID owners (if they exist) */ SBMBinary struct IDList global protected ? owningSymbol.q, count.l, global packedIDs ? {firstID.l, check:3 = 0, depth:5, firstID.l, id:depth[leading idCount], align.b}:{id.l[leading idCount]}; /* Original (old) Dumbit HyperTalk version from the ÒLyricsÓ stack -- which... never got anywhere, and is obsolete Stages were originally called ÒSheetsÓ - contained in ÒPagesÓ. -- tag.q, offset.l, idCount.l -- struct idList firstID.l, global packedIDs?{clear:3, depth:5, id:depth[leading idCount]}:{id.l[leading idCount]}, bytealign; put hextonum(char 1 to 16 of chunk)into tag put hextonum(char 17 to 24 of chunk)into nextID put hextonum(char 25 to 32 of chunk)into idCount -- minus one put 128 into pos repeat while(t""and x