Problem Statement: I/O performance ....database queries, performance ...web hits ...web-server-responses ...back-end storage, peformance optimization by introducing one layer where performance can be maximized or optimized by having a appropriate [dynamic] storage provisioning(storage virtualization) ...viz. no of I/O hits, storage space requirement(lun size or filesystem size) further as published in the document projects.pdf ( 175), 182), 184), 187),253), 254)!!, 426),591),798), and else where by this individual. !! one of the reason's why RAID and storagemanagement is better-off automated is , in a one off a kind of assignment i took-up in the recent past ....a fairly accomplished sysadmin with 15 - 20 yrs of experience on unix systems ....when monitoring and managing lun's ...is counting up 'iostat' stat's on virtual lun's carved out of a RAID ...storage-array!! ...precision ...ability to armour ... a sysadmin with kind of ability fine tune ....performance!! engineering!! ....given that storage forms the foundation based on which the performance of the most application's host or making use of storage-space from such raid solutions. !!, while a iostat can still reveal the fact that RAID solution or virtual lun is at best skewed. virtualize RAID, volume-manager, virtual file-systems, serveral other use-case modelled functionalities at lun level ....in high-end storage box's or array's / SAN environments ...this is different or extends beyond what was or is the functionally feasible on storage array's like sun a1000 , a5000 and also effectively utilize the processing power. Automation inside the storage-box-array/storage behind SAN-fabrics by encapsulating innovative set of layer's of software viz. RAID/volume-management/filesystem-management/use-case-modeled...software stack...compression/backup...etc Opportunity a:) Refer to the document opportunity_innovative_web_token_session_mgmt.txt, opportunity_webapplication_performance_tuning.txt, opportunitydbenhancementsperftuning.txt, By understanding the computing virtual architectures(2tier or 3tier ...differentation in various layers or categories in software stack viz. application-processing ...databse-processing), load balancing of traffic can be done at applications-layer ( both applications layer ...websevers, email, group-ware ,file, database servers) ...a hidden layer can be introduced ...which at storage layer...where I/O through puts can be optimized. by having the storage for webservers, database servers ...with RAID ...and volume-manager for storage managing storage in high-end storage-arrays or storage-area-networks ...with the servers provisioned with sufficient bus-width/bus-speeds or bandwidth ...to these storage. . Opportunity b:) Performance engineering of IO ...metric's ...efficiency based the usage of the lun viz. optimized for writes / reads deriving from RAID ...mirroring /concatenation ...a concepts used in volume-manager embedding or bundling the concepts of RAID with highend-storage-arrays or SAN..., ...luns allocated to a host can be managed virtually ...based on the I/O hits , data-usage. where in the embedded software layers in storage-arrays or SAN determine the layout as also architecture of the VIRTUAL lun based on the RAID principles, [filesystem management] Eg: based on the I/O hits ...additional mirrors can be provisioned in SAN/Storage-array, based on the present space(empty...expansion principle) requirement ...can be provisioned dynamically by (...allocating a larger virtual lun) or ( by changing the layout of the lun using RAID) . while raid 1+0 with number of mirrors increased proportional to i/o hits or traffic offering best layout, simulate raid 5 with mirror's for all disk participating in the raid 5 , 1+0 for the parity disk in raid 4, choosing the best layout...also offering redundancy based on i/o and redundancy requirements ----------------------------------------------------------------------------------------------------------------- Ability to implement RAID on a bunch's of mirrored disks (vi. each disk participating being virtual disk) ...interpret [ analyze ...vxiod daemon in the vertias suite ...buffer the io to multiple disks simultaneously...multiplexing] ------------------------------------------------------------------------------------------------------- ======================================================================================================= thus the scope of implementing raid 5 or raid 4 on a mirror ...given the scope or context that io can be buffered to multiple disk's at the disk driver level. a disk being the basic unit or building block in standard raid definition, the concept of building block for raid i.e. disk can be redefined ...thus a virtual disk (1+0 or 'n' number of disks as required for redundancy and performance) can be virtuall defined as single disk a bulding block and used in raid layout's. a different kind of definition for ...what has traditionally been defined for plex or out look implementing the raid and plex's ...as can be defined in raid concepts ...and taking veritas raid implementation or definitions as a anology ...refer to how ods in older versions of solaris is implemented ...the same envisaged at the level of a disk as the basic minimum building block instead or inlieu of a disk partition....to leverage full potential of the RAID ....viz. raid 0 being to do with disks ...ability to bunch disks ...buffer the io on a virtual disk, to leverage the capabilities of the RAID and its various kinds of implementations. example scenario being potential ability to implement a radi 5 on 3 (three) a1000 or a5000 arranged or connected to a OS host in series. ------------------------------------------------------------------------------------------------------------------ Opportunity c:) further to opportunity b) ...a file-system could also be virtualized on the storage allocated to hosts , by shielding the OS layer ...viz. data-management of all applications ,...say based on attributes settable by OS ...RAW or filesystem ....viz. by bundling or embedding the filesystem management principles or logic with SAN/Storage-array management logic boards or embedded OS, with virtual luns marked for filesystem management. a notable managable-attribute that can be set ...viz. when using a filesystem-lun ...space to managed (defragment or re-organization of contigious-space allocation) by the host to which lun(say kind of database) is allocated or array managed...a key factor for database performance tunining [ ....by utilizing concepts of mirroring ...by potentially having a attribute 'quiecesing' interval/time-at-which configurable through a management interface on the host to which the lun was allocated deriving from how a bit based archiver or differential (dirty-page logging) ...efficiency in backups, how snapshot's work ....a virtually defragment lun introduced into a mirror ...which can be brough-online and other copies in the mirror synchronized or where no provision for mirroring ...the non-defragmented lun can be taken offline. ] ...given the scope of the virtual space in virtual file systems (optionally veritas) ...how virtual contigous address space is generated ...viz. i/o optimization ....virtual address generation of the data or rows in the filesystem for optimal tablescan i.e. ability to use concepts used b-tree or index or data hashing alogorithms ...as part of the virtual address space generation in filesystem ...so as to max or leverage the advantages of b-tree loading of the node or data retrival....refer to 'opportunity_innovativefoundation_4_ERP_MIS_accounting_applications.txt' and 'opportunity_dynamic_virtualization_storage_blueprint.txt' eg:refer to the context ... 'opportunity_database_optimization_tuning_tip.txt' Opportunity d:) ability to increase the processing power of SAN/Storage-arrays ...given that these are appliances , caterin to specific needs ...thus the ability to add additional processing power by hot swappable or pluggable processors ...much like in high-end computing arcitectures ...given the context of 1) Manage more no of virtual-luns , hard-luns ...additional storage provisioned into a SAN/storage-array. 2) refer as specified in Opportunity b) & opportunity c). opportunity e:) with reference to opportunity b) and opportunity c) as mentioned above ...a protocol or standard ...viz. based on xml-rpc or com/dcom or corba ...which gives the application software ...viz. mail-servers/ database-servers/group-ware/ ...a interface to interact with embedded-software-layer containing or maintaining RAID and filesystem-management on the luns being used by their specific host for exchange data specific to metrics and storage....viz. requirements like provisioning of a lun , rawspace or filesystem-space, preset no of mirrors for I/O balancing or a host communicated dynamic number of mirrors for a lun allocated to a host ...over-ridable or not over-ridable, [ backup-frequency of the lun, snapshot ...frequency of the lun ...refer to projects.pdf & else-where as envisaged by this individual , remote replication of lun ...say compatible appliance installed with sufficient bandwidth ], ...use-case modelled list of interfaces, and functionalities to be implementable. while it may be remembered that each virtual-lun mapping to at least one physical-lun i.e. hard-disk gives ideal or optimal results ...as sharing a physical-lun by one or more virtual luns can potentially result in I/O hindrance. Opportunity f:) with management interface ....ability to enable/disable by admistrator/interface that lets these attributes to be managed or set at the OS or application level to which a lun is allocated from the storage box ... various attributes applicable or configurable at lun level. apart from below ...this include's ...instead of a storage administrator carving out a lun and allocation to host, the management interface on the host ability to requisition or provision for a lun of specific size to the storage-array/SAN, which in effect also keep's or take's into account as discussed in b) above. eg: content versioning ...could potential be made part of such storage-box or array with addon logic-card or customizable feature(optionally loadable kernel module) on such a storage box or container. specific scope for feasibility being deriving from how compression works ...unix...diff ...utilities ...efficeint storage and organization of content on such luns. also given the context of patented concepts like bit-based archiver / bit-based flash-backups(plex or volumes) as also data backup to tape mediums...superior backup solutions (may refer products and marketing literature for backup solutions from ibm,veritas,legato ) etc. Deriving from how hardware-compression is handled in storage backup solutions, similarly ...[ a generic-compression-card ]...can be part of the storage-box, that can be utilized ...when a attribute on a lun is enabled for data compression. refer to the document 'opportunity_DWH_BI_product_orchrestrization.txt' Opportunity g:) deriving from how grid computing works (eg: sun fire servers, e10k....ibm lpars...) ....a innovative storage-box or container or array deriving from above scenarios or business opportunities ....a dynamic algorithm for grid computing ...that allocates sufficient processing power and resources to a specific task ....applying use-case modelling makes the envisioned sketch ...into a feasible solution. Eg : usage case scenario or potential scope of above envisaged solutions file-servers as apache+http_fileserver over volume-manager + [virtual file management solutions] highend storage in back ground using iscsi or gigabit-ethernet , storing or recording GPS data-streams or GPS feeds, similarly recording stock market feeds also refer to opportunity_innovative_web_token_session_mgmt.txt, opportunity_integrated_EDI_fax_scanner_portable_computing.txt, opportunity_datastorage_mediums_data_access_views.txt, opportunity_innovative_directory_servers_views.txt. Opportunity h:) deriving from opportunity c) and other context above ...given the scope of the virtual space in virtual file systems (optionally veritas) ...how virtual contigous address space is generated ...viz. i/o optimization ....virtual address generation of the data or rows in the filesystem for optimal tablescan i.e. ability to use concepts used b-tree or index or data hashing alogorithms ...as part of the virtual address space generation in filesystem ...so as to max or leverage the advantages of b-tree loading of the node or data retrival....refer to 'opportunity_innovativefoundation_4_ERP_MIS_accounting_applications.txt' and 'opportunity_dynamic_virtualization_storage_blueprint.txt' Opportunity i:) deriving from all of the opportunities as interpretable or envisable from above read through the document "opportunity_client_side_server_side_validation_storage_into_repository.txt" ability to provision or usage operating-system-kernel-module for processing of data, before storing into the storage. Opportunity j:) Deriving from grid-computing other similar architecture's in vogue, much like the ability to allocate more processing power to process'es requiring more processing power ...say analogous to thread priority, ability to allocate more disk-block's (mirror's ...at a microscopic level that a lun) to logical data-block's (or portions of a file-system) impacted by high IO ...without having to re-organize underlying physical-disk-layout for the logical data-blocks-or-structure to achieve efficiency or performance in IO. A filesystem is nothing but hashing of data on a data with a user-friendly api ...access such data. A good understanding of Raid 0 (stripe size), virtual filesystems (vxvm), lvm, ods, vxvm, ability to re-architect how disk IO is done at disk-driver level , embedded-firmware of storage box's. Read through opportunity_datastorage_mediums_data_access_views.txt Problem statement : Deriving from various realtime use-case scenario's ...ability to arrive at fine-grained control over the number of mirror's ...splintering or bifurcating IO's .....contagious....virtual-address-space. case-study: studying the disk-io blockage or hindrance of a DB2 instance on raid over scsi-disks in a ds4300 array a 2000 odd users accessing ...a 3 tier application ...with majority of hits effecting 3-4 database objects, ability to interpret and control some or more of the following ...the usage or interpretation of efficiency from database i.e. data-structures of organization of data while a physical contagious disk space ...can give efficient data-reads ...when implementing the concept of striping ...ideally translate into efficient IO read's ...simultanous reads from disparte disks ....ability read more data than from a single disk ...ability to leverage caching or physical-memory(RAM) for efficiency. a Qmail application catering to 1500+ users on a single scsi-25-GB disk with a 5mb storage-folder-each, with majority or concentration of the users in a few-specific folder's ...thus ability to identity cause for IO hits or blockage...ability keep the logical-structure of the data intact and force a physical placement a few logical structure entities(containers or directories) across spindles or relocate, control the physical data placement. how symbolic-links work ...thus the ability to chain disk blocks spread across the disks, innovation in linking chunks of disk blocks ...understanding unix process communication architecture ...pipes...descriptors, ability mirror ...disk-blocks ...that can fit in the given scope of the context detail in this section. Disk I/O is constraint ...not envisaged at most application software level in the stack of software. ability to gain fine grained control over placement of folder/containers/directories on virtual-filesystem. Irrespective of the capacity of disk , RPM rate's offered ....given that number of thread's /sessions seeking IO on the disk ...i.e. higher-level ...number of concurrent user's who can access data, use application-software without performance hindrance. given that above is the case-scenario most application-software in the stack above a disk IO driver, wether email-application, or database. While having or following the advocacy of having a RAID solution with more mirror's would solve the performance-hindrance to a extent. Load-test-scenario 1: for instance qmail servicing SMTP , data hosted on the server with a single disk ...scaling up of users 2: given that databases require contigious disk-space for efficienct IO, data oraganization ...number of requests from end-users to a object in database or number concurrent read/write's with-out hindrance to disk IO performance. addressing the scenarios in 1) , 2) ...by employing RAID solution judiciously. ability to generate virtual-lun /virtual-filesystem ...by having the ability to define the number of mirror's, as also having the data spread across virtual-address-space i.e. interleaved or spliced or striped , ...by utilizing or deriving from the lvm(for instance aix/hp,ods-solaris,vxvm) concepts ....ability to define more number of mirror's to data-subsets (eg: in the filesystem or a database ...viz. a container or folder or directory ....given that specific data-sets experience's more number of IO hits, instead of having a homogenous i.e. uniform ...mirroring-level across the lun. feasibility given the scope , being subject to disk-geometry ...io plexing ...various attributes of disk's, arrays, putting place appropriate disk i/o device-driver at host level or embedded firmware in array other similar storage-box's, matching the objectives. ----------------------------------------------------------------------------------------------------------------------------- Given that most the RAID carving or derivations, virtual-file-system are virtual entities in the sense of logical interpretability or tangible nature of their sense. the below envisages essentially how data should reside ...and a logical view of such data ...as a fallout of potential advantanges or objectives to achieved ...keeping the integrity of the logical view of the data, writes on disk translating into data consistency and integrity ....traversal of filesystem. rationalizable concepts handing few concepts, trade-offs, advantages (mirroring) at logical lun-level or chunks of file-system impacted by heavy io ....ideally all objectives envisage above should be in a automated manner or a high-level-advanced user or administrator of such data-repositories. Eg: deriving from how meta-data is stored in directories , super-blocks ...re-architecting the structure of meta-data, as also understanding how file's, folder's are linked in a file-system ....how various file-systems in vogue on various operating-systems have crash-proof or recovery of data is achieved, how fsck works or is implemented, in most filesystems only superblocks are replicated or duplicated or multiple copies present on the file-system for redundancy .... A customized or custom developed filesystem where data-blocks are also duplicated or replicated or multiple copies stored on the filesystem ...such that as envisaged various objectives defined above are fulfilled for instance when traversing a file-system ...from one data-block to next-data-block ....the addressing schema may be such that when traversing from one datablock to next data-block ...the pointer from data-block may contain or traverse to any of the copies(mirror's) ...of the subsequent datablocks. conceptualizable idea: Begin by breaking the physical-disks into addressable optimized splice-size or block-size for or analogical concept: a ideal IO blocking factor....when implementing a address schema for a disk-space ...standard ufs contigious-disk-space(viz interpretations of a disk-io seek() call) being the only means. data-hashing ...using a filesystem ...implemented as a hierarcial-tree ...at each node ...branching into multiple-child-nodes ...at any given node-level representing data or copies of mirror's ...ideally carving into or disk-space on disk's or platter's across spindles, i.e.at any given node-level all child-nodes at the level are mirror-copies or themselves located on disparate disk space. Making directories or container's as analogous equivalent of a volume ...place's for holding key meta-data...various controlable features. Most file-systems are hirerarcial in nature ...ability to encode or use a relative addressing-schema, much on the lines how hireracy of folders or directories or chained to form a virtual view of a file-system, combining the features of virtual-file-systems where-in a folder or directory or container can grow or add more-space based on requirement, grows dynamically ...instead of standard addressing schema as found in the ufs ... address space always remains or appears contigious ...as a file grows so does its address-space, visualizing how a file IO is done ...traversal from one data block to subsequent datablocks ...ability encode the disk-access-data-blocks also as part of leaf-nodes or leaf-blocks. Note: The above problem statement having been encountered in various scenarios and detailed in various 'Proof of concepts' as mentioned in http://uk.geocities.com/ravivenkatus/projects.pdf http://ravishankarkv.tripod.com/projects.pdf ....apply appropriate 'use-case' modeling, rationalize and arrive at a workable and feasible solution both commercially and techinically viable.