KiXforms Forum Index KiXforms
The Forum for the KiXforms Community
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 
 Quick Links 
Site News
Script Archive
Tracking Systems

Building large projects

Post new topic   Reply to topic    KiXforms Forum Index -> Script Archive
View previous topic :: View next topic  
Author Message
KiXforms Regular
KiXforms Regular

Joined: 07 Mar 2003
Posts: 41
Location: Mahwah, NJ

PostPosted: Sun Jan 11, 2004 1:07 am    Post subject: Building large projects Reply with quote

I'm working on a large KixForms project (currently 200K source). These projects can get pretty unwieldly to manage, so I have some code and processes to help me manage it.

This isn't a KixForms script, but it works especially well for building KixForms projects, so I'm posting it here. I'll post the large project I mentioned above shortly, once testing is complete.

Basically, I have a KixDev folder on my PC. In this folder, I have a KixLib subfolder and several project subfolders. The KixLib folder contains all of my general purpose KIX functions - one or two functions per file. (for personal sanity, I name the ones I write with .KXF extensions, and the ones I download with .UDF extensions, but that's up to you.)

Now, when I start a project, I create a project folder. The main project - usually just a header and comments, is in an appropriatly named .TXT file. Project specific function files, along with files that represent each major collection of forms declarations. I do take care to prefix my file names with FRM## and PRG## so the files are loaded in a sequence I can control - forms first, then functions. (the project TXT file is always first)

When I've completed my collection of individual project files, I run KGEN project - specifying the name of the project TXT file. This causes the following actions to occur:

    The name of all functions defined in the files in the KixLib folder are identified.

    The project file and all of the .UDF files in the project folder are then scanned for external functions.

    Those function files are then scanned to resolve dependencies.

    An output file is generated from all the (UDF) files in the project folder.

    All required external library files are added to the generated file.

    Optionally, comments are stripped from the generated file to reduce its size.

Using this process, my large project is made up of about 30 small files - most are only 2-3K each.

For large projects, I define a BUILD.INI file that has one section and 4 entries, similar to the one below:


#Name of project.txt file
# Extension of generated file (.KXW = kixforms script)
# copy to alternate files or directories
# strips comments if defined - retains .GEN file with comments

Here's the KGEN script itself:

;; kgen - generate a kix script, automatically locating and including required UDFs
;; Glenn Barnas - FRIT/EROC   (
;; Version 1.1 - May 21, 2000
;; Fix          - Jan 07, 2004 - Updated to include ALL local project files, even
;;                              if they don't contain functions (form defs, etc)
;; Usage: kgen project_name
;; Locates all UDF files in the current and library folder & identifies actual
;; udf name, creating a function to file map. Once this is complete, the project
;; file is scanned for any of these functions, and those functions are then scanned
;; for other UDFs. The resulting list of UDF names and files is then sorted and
;; duplicate filenames are removed. This final list of required UDFs is then combined
;; with the source project file in a new output file. Note that the scanning of UDFs
;; for other UDFs IS RECURSIVE!
;; The source file is standard KIX code and can freely reference any UDF contained in
;; the current directory or the common library (defined here as ..\KixLib). Large
;; projects can be broken into smaller pieces and stored in UDFs in the current
;; directory. UDFs that are more generic are stored in a common library. The UDF
;; search path can be customized by adding additional paths to the PATHS variable.
;; The source file must use a standard .TXT extension to avoid confusion with the
;; script output file (.KIX) and project-specific UDFs.
;; The command searches for all files in the library locations that end in "f". This
;; is done because our organization uses .KXF for KiX Function libraries in production
;; environments, and I use .UDF on the dev library to distinguish between "internally"
;; and "externally" developed functions.
;; The kix development structure that this tool supports is as follows:
;; KixDev
;;    +--KixLib
;;    +--Project1
;;    +--Project2
;;    +--Projectn
;; From a project folder, run KGEN ProjectName to "compile" the UDFs into a single script.
;; Note that this tool does not detect functions that are called but not defined
;; either within the source file or in an external UDF!
;; The base project name, output file extension, and list of additional directories to
;; copy the project file to can be specified in a BUILD.INI file in the project folder.
;; This allows KGEN to be run with no parameters. There is one section - [Project] - with
;; three keys - Base, Extn, and Dirs. Base is the basename of the project file; Extn is
;; the extension of the generated script (default=.kix); and Dirs is a ';' separated list
;; of paths that the generated script should be copied to. This is a convenient way to copy
;; the script from the project folder to the test environment automatically.
;; KGen has the ability to strip comments from code to reduce the script size (or obfuscate
;; the process?) by specifying a second argument (anything) on the kgen command line.
;; This can also be accomplished via the build.ini file with a STRIP=1 directive.
;; a string, that line MUST have a comment character at the end of the line or the line will
;; be corrupted! For example: ( $X = ";" ; ) is OK, but ( $X = ";" ) is not.
Break On
Dim $Functions[250,2]
Dim $FnFiles[100]
Dim $File, $Tmp, $P, $FP, $FC, $OP, $FnList, $Ext, $OutDirs, $Dir, $CF, $Parsed, $LoadType
Dim $SrcFile, $OutFile, $LogFile, $CMsg, $Strip, $Line
; list of directories to scan for UDFs - "." is REQUIRED
If %KixIncludePath% <> ""
  $PATHS = ".", "..\KixLib", %KixIncludePath%
  $PATHS = ".", "..\KixLib"
; Should UDFs outside of the project folder be Embedded or Referenced via Call statement?
$LoadType = 0                   ; 0=Embedded, 1=Referenced
; Default extension for generated script (usually .KIX for text and .KXW for KixForms)
$Ext = '.kix'                   ; default output extension
$FP = 0
$Parsed = ''                    ; list of files parsed
; If BUILD.INI exists, get the base file name and output extension values.
If Exist('build.ini')
  If $A1 = ''
    $A1 = ReadProfileString('.\build.ini', 'Project', 'Base')
  If $A2 = ''
    $A2 = ReadProfileString('.\build.ini', 'Project', 'Strip')
  $Tmp = ReadProfileString('.\build.ini', 'Project', 'Extn')
  ; If output extension is defined, override default of .KIX
  If $Tmp <> '' $Ext = $Tmp EndIf
  $OutDirs = ReadProfileString('.\build.ini', 'Project', 'Dirs')
$SrcFile = $A1 + '.txt'         ; project source file name
$OutFile = $A1 + '.gen'         ; Generated output file name
$ScrFile = $A1 + $Ext           ; project output file name
$LogFile = $A1 + '.log'         ; project log file name
; make sure that a name was specified
If $SrcFile = ''
  ? 'KGen: build file not specified'
  ? 'KGen: Aborting!' ? ?
; Make sure the specified build file exists
If Not Exist($SrcFile)
  ? 'KGen: Source file ' + $SrcFile + ' was not found in the current directory'
  ? 'KGen: Aborting!' ? ?
; Start by deleting the output files
Del $OutFile
Del $LogFile
; Assemble a 2-dimensional array containing a list of function names and the files
; that hold them. Search the folders defined in the PATHS array
'Generating' + Chr(13)
For Each $PATH in $PATHS
  ; Look for files that end in "F" (.KXF and .UDF)
  $File = Dir($Path + '\*.??F')
  While $File <> '' And @ERROR = 0
    ; open the file, read lines until 'function' is found in a non-comment area
    ; obtain the function name, and add the function name and file name to the array
    If Open(3, $Path + '\' + $File,2) = 0
      $Line = ReadLine(3)
      $Tag = 1
      While @ERROR = 0 And $Tag = 1
        $Line = Split($Line, ';', 1)[0]
        $P = InStr(Trim($Line), 'Function')
        ; Function definition must be in column 1 after trimming spaces to be considered valid
        If $P = 1
          ; get rid of the 'function ' and split off the function name
          $Line = Split(Right($Line, Len($Line) - 9), '(', 1)[0]
          $Functions[$FP,0] = $Line
          $Functions[$FP,1] = $PATH + '\' + $File
          $FP = $FP + 1
          $Tag = 0
        $Line = ReadLine(3)
      Loop ; while not error
      ; Tag will be 1 if no functions are defined in the file. If the .KXF/.UDF file
      ; is in the project folder (.\) it should be included anyway!
      If $Tag = 1 And $Path = '.'
        $Functions[$FP,0] = '_FaKe_'
        $Functions[$FP,1] = $PATH + '\' + $File
        $FP = $FP + 1
    $ = Close(3)
    $File = Dir()
Next ; $Path
; We now have a list of available functions in the library, and the files that contain them.
; The base file specified must be scanned to identify which of these functions are referenced,
; and then those functions must be scanned to determine if any dependencies to other functions
; exist.
$FP = $FP - 1   ; reset the upper limit
$OP = 0
$FnList = ''
; "preload" the function list with all UDF files from the project folder (.)
; These UDF files are always embedded with the base file. They are loaded in
; alphabetical order, so start with numbers to control the sequence.
Gosub PreLoadFn
; Now check the files to determine which external UDFs must be included
; Open the source file and parse it for functions
$InFile = $SrcFile
Gosub FindFn
; Check each of the required function files for dependencies on other functions
$CF = 0
For Each $InFile in $FnFiles
  Gosub FindFn
; If the change flag is set, process the newly added UDFs for additional UDF references
If $CF = 1 Goto 'Scan' EndIf
'                                           ' + Chr(13) ; clear the message line
If $OP = 0
  "No UDFs needed for this generation!" ?
  Goto 'BUILD'
; reduce the array size
ReDim Preserve $FnFiles[$OP - 1]
; sort the array
$FnFiles = QSort($FnFiles)
; remove duplicates
$FnFiles = Uniq($FnFiles)
; If $LoadType <> 0, the referenced external UDF files must be "call"ed first
; this is not yet supported, awaiting a reasonable, universal method
; One way is to start the script with a "gosub _Load_External_UDFS_", and then
; surround the generated call statements with the label and a RETURN statement.
; This isn't "elegant", but putting all the external call statements at the top
; of the generated file may not be pretty either.
; Assemble the output file ($OutFile) from the base and all necessary includes.
$ = RedirectOutput($LogFile)
? 'Building $OutFile from:' ?
; start with the project source file
$CMsg = ';; KixGenerated: @DATE - @TIME @CRLF'
$File = '.\' + $SrcFile
Gosub GenFile
$CMsg = ''
; Add all of the include files
For Each $File in $FnFiles
  Gosub GenFile
; Now have a .GEN file - possibly strip comments and generate script file
$Strip = IIf($A2 <> '', 1, 0)
; Delete the script file
Del $ScrFile
$ = Open(4, $OutFile, 2)
If @ERROR = 0
  $ = ReDirectOutput($ScrFile)
  $Line = ReadLine(4)
  While @ERROR = 0
    $Comment = InStr(Trim($Line), ';')
    If $Strip And $Comment
      If $Comment = 1
        ; skip comment line
      Else   ; could attempt to remove comments from right-end of line
        ;$Line   ; (or not!)
   $Comment = InStrRev($Line, ';') - 1
   RTrim(Left($Line, $Comment)) ?
      $Line ?
    $Line = ReadLine(4)
  $ = Close(4)
  $ = ReDirectOutput('')
; Generation is complete - copy to secondary folders if defined
If $OutDirs <> ''
  $OutDirs = Split($OutDirs, ';')
  For Each $Dir in $OutDirs
  Copy $ScrFile $Dir
'Generation of $ScrFile is complete!' ? ?
; SubRoutines ==========================================
; subs use global vars for simplicity in maintaining state
If InStr($Parsed, $InFile) > 0 Return EndIf
If Open(3, $InFile,2) = 0
  'Examining $InFile                 ' + Chr(13)
  $ = RedirectOutput($LogFile)
  'Examining $InFile                 ' ?
  $Line = ReadLine(3)
  While @ERROR = 0
    ; Get the non-comment line (portion)
    $Line = Split($Line, ';', 1)[0]
    For $FC = 0 to $FP
      $Fn1 = $Functions[$FC,0] + '('
      $Fn2 = $Functions[$FC,0] + ' ('
      If InStr($Line, $Fn1) > 0 Or InStr($Line, $Fn2) > 0
        If InStr($FnList, $Functions[$FC,0]) = 0
          $FnList = $FnList + $Functions[$FC,0]
          $FnFiles[$OP] = $Functions[$FC,1]
          ' adding function ' + $Functions[$FC,0] + ' from ' + $Functions[$FC,1] ?
          $OP = $OP + 1
          $CF = 1 ; set ChangeFlag to force rescan
      EndIf ; instr fn
    Next ; $FC
    $Line = ReadLine(3)
EndIf ; open
$ = Close(3)
$ = RedirectOutput('')
$Parsed = $Parsed + $InFile
; Insure that all of the UDFs in the project directory are loaded, even if they haven't been referenced
; This allows KGEN to generate larger library files
$ = RedirectOutput($LogFile)
'Loading project UDFs' ?
For $Fc = 0 to $FP
  If Left($Functions[$FC,1],2) = '.\'
    $ = RedirectOutput($LogFile)
    $FnList = $FnList + $Functions[$FC,0]
    $FnFiles[$OP] = $Functions[$FC,1]
    $OP = $OP + 1
    ' adding function ' + $Functions[$FC,0] + ' from ' + $Functions[$FC,1] ?
$ = RedirectOutput('')
; add the named file to the generated script file & update the log
  ; send output to the destination file
  $ = RedirectOutput($OutFile)
  If Left($File,2) = '.\' Or $LoadType = 0
    ; send output to the log file to describe the build
    $ = RedirectOutput($LogFile)
    '  $File' ?
    ? "Call '$File'" ?
    ; send output to the log file to describe the build
    $ = RedirectOutput($LogFile)
    '  Call $File' ?
  ; Output back to the screen
  $ = RedirectOutput('')
;;FUNCTION       qsort() (originally "qs()")
;;ACTION         Sorts a 1-dimension array
;;AUTHOR         "BrianTX"
;;SYNTAX         qsort(array)
;;PARAMETERS     array - array to be sorted
;;REMARKS        Sorts numeric or text (ASCII order) arrays
;;RETURNS        array, sorted
;;EXAMPLES       $Sorted = qs($Unsorted)
Function qsort($a)
  DIM $ls[32],$us[32],$sp,$L,$U,$m,$p,$i,$j,$t
  $ls[0] = 0            ; Lower Stack Index
  $us[0] = UBOUND($a)   ; Upper Stack Index
  $sp    = 0            ; Stack Pointer
  While $sp >=0
    While $L < $U
      While ($i<$j) AND $A[$L] > $A[$i]
      While ($j>=$i) AND $A[$j] > $A[$L]
      IF $i >= $j goto L2 ENDIF
      Goto L1
      If $m-$l <= $u - $m
        If $m+1 < $u
        If $m-1 > $l
;;FUNCTION       uniq()
;;ACTION         Removes duplicates from a sorted, 1-dimension array
;;AUTHOR         Glenn Barnas / FRIT-EROC
;;SYNTAX         uniq(array)
;;PARAMETERS     array - array to remove duplicates from
;;REMARKS        Array must be sorted first - no check for sort is performed
;;RETURNS        array with duplicate items removed
;;EXAMPLES       $Singles = Uniq($has_dups)
Function Uniq($A)
  ; $TOP is last entry, $X is source pointer, $Y is destination pointer
  Dim $Top, $X, $Y
  $Top = UBound($A)             ; Find size of original array
  Dim $WA[$Top]                 ; Create working array
  $Y = 1                        ; output array pointer
  $WA[0] = CStr($A[0])          ; copy first element
  For $X = 1 to $Top
    If $WA[$Y - 1] <> $A[$X]    ; is current value different from last?
      $WA[$Y] = CStr($A[$X])    ; add it to output
      $Y = $Y + 1               ; increment output pointer
  ReDim Preserve $WA[$Y - 1]    ; resize output array
  $uniq = $WA                   ; return the array of unique elements

I use a simple KGEN.BAT file to invoke this, passing the arguments to the kix script:

@Kix32.exe kgen.kix $A1="%1" $A2="%2"

OK - teaser! Here's a screenshot of the large (13,000+ lines) project I'm working on...

Back to top
View user's profile Send private message Visit poster's website
KiXforms Regular
KiXforms Regular

Joined: 07 Mar 2003
Posts: 41
Location: Mahwah, NJ

PostPosted: Wed Dec 02, 2009 1:19 am    Post subject: KixDev - KGen+Sanity Reply with quote

This project has evolved into the KixDev package, which is built around KGen. KGen resolves UDF dependencies and combines multiple package and UDF files into a finished script. It can then optionally tokenize the resulting script, and copy it to other locations.

From a developer standpoint, it generates several reports on variable usage (declaration, first use, undeclared, declared but unused) and code structure. It performs a sanity check on the resulting script, looking for mismatched quotes, parenthesis, and paired functions such as If/Endif and While/Loop.

We use KGen internally for all of our Kix and KF script development.

Version 2.4 is available for download from my web site - - in the Products / Admin Toolchest section. This is just one of the free tools we offer to the Kix development community. The ZIP package includes the KGen script, a PDF User Guide, and a copy of our entire development UDF library.

Back to top
View user's profile Send private message Visit poster's website
Display posts from previous:   
Post new topic   Reply to topic    KiXforms Forum Index -> Script Archive All times are GMT
Page 1 of 1

Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum

Powered by phpBB © 2001, 2005 phpBB Group