KiXforms Forum Index KiXforms
The Forum for the KiXforms Community
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 
 Quick Links 
Site News
Downloads
Documentation
Donations
Script Archive
Tracking Systems

LogViewer Script - Request for optimization

 
Post new topic   Reply to topic    KiXforms Forum Index -> KiXforms.NET Discussion
View previous topic :: View next topic  
Author Message
mercury
KiXforms Regular
KiXforms Regular


Joined: 10 Aug 2005
Posts: 50

PostPosted: Sun May 21, 2006 6:45 pm    Post subject: LogViewer Script - Request for optimization Reply with quote

Hi All,

I thought it was about time I started 'giving back' and share some scripts I've been working on.

This first one is a utility I've been using to read log files generated by my other scripts. The log files are simply tab separated text files that I would usually load into notepad or excel to view. As these files grew larger in size I wanted a quicker way to view and filter the results. I could have written some sort of macro in Excel but then I would also need to have Office loaded on every machine I needed to view the logs on - not something I really want to put on a server. Therefore this provided just the excuse to write something in KiXtart/KiXforms.net

It's been a good learning experience and the script does the required job but I believe there is much room for improvement. I would really welcome some assistance in optimizing the code with the hope to achieve closer to Excel speed if possible.

The zip file includes 3 files - The main script and two sample log files.
The last UDF in the script is a copy of the function used to generate the log files. I haven't submitted this to KORG yet as I want to make sure all is are dotted and ts crossed first. Hopefully some of you might find this useful.

Known issues:

*Large files (3000 rows) take a long time load and format.
I am using an array of arrays to load the data and then apply any filters to that array before populating the ListView. I tried multi-dimensional arrays at first but these proved too slow with even modest sized files and there was all sorts of other weird behaviour.

*I would like to know how to specify a default folder for log files so I don't have to keep navigating from the root of the drive.



[/img]



LogViewer.zip
 Description:

Download
 Filename:  LogViewer.zip
 Filesize:  29.02 KB
 Downloaded:  728 Time(s)

Back to top
View user's profile Send private message
Shawn
KiXforms Developer
KiXforms Developer


Joined: 22 Feb 2003
Posts: 1983
Location: Canada

PostPosted: Mon May 22, 2006 1:44 am    Post subject: Reply with quote

Extremely well done. Great interface - intuitive and easy to use.
Back to top
View user's profile Send private message
mercury
KiXforms Regular
KiXforms Regular


Joined: 10 Aug 2005
Posts: 50

PostPosted: Mon May 22, 2006 8:41 am    Post subject: Reply with quote

Thanks Shawn, That means a great deal!

I should point out this part of the ShowLogFileOutput function:

Code:
  ; Add last 62 entries (1280 x 1024)
  $Maxlines = 61 
   $TotalMaxlines = Ubound($CurrentLogFileArray)
      
   If $TotalMaxlines > $Maxlines
      $TotalMaxlines = $Maxlines   
   EndIf


I put this in here in an effort to speed up the app with the assumption that I would mainly want to see recent log entries rather than the whole thing. The entire Log file is always loaded into the $LogFileArray. Any filters specified are applied to this array and a second $FilteredLogFileArray is generated and provided as the source array to the ShowLogFileOutput function. The Progress bar is called at each stage so you can get an idea how long each process takes to complete.

If you change $MaxLines to a vaule of say 3000 and load the NetworkLogonHistory(AA Test).log file you will see how slow it is to populate the ListView with the entire file. Is there a faster alternative? Maybe the ListView is not the right control for this and the DataGrid/View a better option in the future?
Back to top
View user's profile Send private message
Shawn
KiXforms Developer
KiXforms Developer


Joined: 22 Feb 2003
Posts: 1983
Location: Canada

PostPosted: Mon May 22, 2006 10:56 am    Post subject: Reply with quote

yeah, it takes 25 seconds to load on my laptop. Have you tried benchmarking bits of the script to see where things are bogging down ? I will give a try.
Back to top
View user's profile Send private message
mercury
KiXforms Regular
KiXforms Regular


Joined: 10 Aug 2005
Posts: 50

PostPosted: Mon May 22, 2006 11:58 am    Post subject: Reply with quote

According to the attached log it takes about one full second per 126 rows of data to display on my machine (pentium-M 1.7GHz Laptop, 1.5GB RAM).

I added this code:

Code:
Function ShowLogFileOutput($CurrentLogFileArray)
   
   Dim $x,$y
      
  ; Add last 62 entries (1280 x 1024)
  $Maxlines = 126
   $TotalMaxlines = Ubound($CurrentLogFileArray)
      
   If $TotalMaxlines > $Maxlines
      $TotalMaxlines = $Maxlines   
   EndIf
   
   ;###
   $StatusBarPanel1.Text = "Displaying Log File..."
   ;###   
   
   $Form1.Cursor = $System.Cursors.WaitCursor
   $ListViewLogOutput.BeginUpdate
   
   $ProgressBar.Value = 0
   $ProgressBar.Maximum = $TotalMaxlines
   
   $StatusBarPanel3.Text = $TotalMaxlines
   
   $nul = ActivityLog("Output Log File", "Start", "Begin Populating ListView", $LogPath)
   
    $ActualRow = 0
  For $Currentrow = Ubound($CurrentLogFileArray) - $TotalMaxlines To Ubound($CurrentLogFileArray)
     $nul = ActivityLog("Create Row: " + $ActualRow, "Start", "Create First Item", $LogPath)
     $nul = $ListViewLogOutput.Items.Add($System.ListViewItem($CurrentLogFileArray[$Currentrow][0]))
     $ActualSubitem = 0
     For Each $SubItem in $CurrentLogFileArray[$Currentrow]
;        $nul = ActivityLog("Create Subitem: " + $ActualSubitem, "Start", "Create each SubItem", $LogPath)
        If $SubItem <> $CurrentLogFileArray[$Currentrow][0]
           $nul = $ListViewLogOutput.Items.Item($ActualRow).SubItems.Add($System.ListViewSubItem($SubItem))
        EndIf
        Select
           Case $SubItem = "Start"
              $ListViewLogOutput.Items.Item($ActualRow).ForeColor = $system.Color.DarkGreen
              $ListViewLogOutput.Items.Item($ActualRow).BackColor = $system.Color.Yellow
           Case $SubItem = "Finish"
              $ListViewLogOutput.Items.Item($ActualRow).UseItemStyleForSubItems = 0
              $ListViewLogOutput.Items.Item($ActualRow).Subitems.Item($ActualSubitem).ForeColor = $system.Color.Blue
           Case $SubItem = "Success"
              $ListViewLogOutput.Items.Item($ActualRow).UseItemStyleForSubItems = 0
              $ListViewLogOutput.Items.Item($ActualRow).Subitems.Item($ActualSubitem).ForeColor = $system.Color.Green
           Case $SubItem = "Fail"
              $ListViewLogOutput.Items.Item($ActualRow).ForeColor = $system.Color.Red
              $ListViewLogOutput.Items.Item($ActualRow).BackColor = $system.Color.Black
           Case $SubItem = "Reboot"
              $ListViewLogOutput.Items.Item($ActualRow).ForeColor = $system.Color.Yellow
              $ListViewLogOutput.Items.Item($ActualRow).BackColor = $system.Color.Gray
           Case $SubItem = "Completed"
              $ListViewLogOutput.Items.Item($ActualRow).UseItemStyleForSubItems = 0
              $ListViewLogOutput.Items.Item($ActualRow).Subitems.Item($ActualSubitem).ForeColor = $system.Color.SteelBlue
           Case 1
              ;
        EndSelect
;        $nul = ActivityLog("Create Subitem: " + $ActualSubitem, "Finish", "Create each SubItem", $LogPath)
        $ActualSubitem = $ActualSubitem + 1
     Next
;     $nul = ActivityLog("Create Row: " + $ActualRow, "Finish", "Create First Item", $LogPath)
     $ActualRow = $ActualRow + 1
     $ProgressBar.Value = $ProgressBar.Value + 1
      $ProgressBar.Refresh
  Next

   $nul = ActivityLog("Output Log File", "Finish", "Complete Populating ListView", $LogPath)
     
  $x = 0
   For Each $Column In $ListViewLogOutput.Columns
      $ListViewLogOutput.Columns.Item($x).Width = -2
      $x = $x + 1
   Next
   
  $ProgressBar.Value = 0

 $ListViewLogOutput.EndUpdate
 $Form1.Cursor = $System.Cursors.Default
 
 $ListViewLogOutput.Items.Item($ActualRow-1).Selected = -1
 $ListViewLogOutput.EnsureVisible($ActualRow-1)
 
   ;###
   $StatusBarPanel1.Text = "Ready"
   ;###
   
EndFunction


As you may notice I originally created a log entry for each subitem and a finish flag but that created a 5MB file with over 58000 rows! My logviewer didn't like that much - I put it out of its misery early Wink
Back to top
View user's profile Send private message
Shawn
KiXforms Developer
KiXforms Developer


Joined: 22 Feb 2003
Posts: 1983
Location: Canada

PostPosted: Mon May 22, 2006 12:12 pm    Post subject: Reply with quote

In my tracing - I am finding the Add(ing) a subitem takes much time. I was playing arounf with defering the adding, kinda like:

$ListViewItem = $System.ListViewItem()

$ListViewSubItem = $System.ListViewSubItem()

; stuff


$= $ListViewItem.SubItems.Add($ListViewSubItem)

$= $ListView.Items.Add($ListViewItem)

If you get my meaning - but your doing kind of a complex load of the listview, not a straight-forward one (i think).

Have another idea about maybe adding an AddRange method to the collections, to speed things up.
Back to top
View user's profile Send private message
Shawn
KiXforms Developer
KiXforms Developer


Joined: 22 Feb 2003
Posts: 1983
Location: Canada

PostPosted: Mon May 22, 2006 12:44 pm    Post subject: Reply with quote

Yeah, working with SubItems is very costly from a processing time stand-point it seems. Searching the net, seems like lots of people have issue with the ListView in this regards.
Back to top
View user's profile Send private message
mercury
KiXforms Regular
KiXforms Regular


Joined: 10 Aug 2005
Posts: 50

PostPosted: Mon May 22, 2006 12:53 pm    Post subject: Reply with quote

So this is more of an inherent .NET thing? - (with ListView subitems in particular)

Are other controls more suited to this?
Back to top
View user's profile Send private message
Shawn
KiXforms Developer
KiXforms Developer


Joined: 22 Feb 2003
Posts: 1983
Location: Canada

PostPosted: Mon May 22, 2006 1:16 pm    Post subject: Reply with quote

DataGrid would be fast, but its not powerfull enough (you cant color cells) - DataGridView would be perfect, buts its not in kf.net.
Back to top
View user's profile Send private message
mercury
KiXforms Regular
KiXforms Regular


Joined: 10 Aug 2005
Posts: 50

PostPosted: Mon May 22, 2006 1:32 pm    Post subject: Reply with quote

oh ok, so when are you adding it? Wink
Back to top
View user's profile Send private message
Shawn
KiXforms Developer
KiXforms Developer


Joined: 22 Feb 2003
Posts: 1983
Location: Canada

PostPosted: Mon May 22, 2006 1:36 pm    Post subject: Reply with quote

Trust me, DataGridView would have been the first control I would have added, if I could have. DataGridView in only available in dotnet v 2.0 - and I cant "go there" right now (for one thing, I dont have the full blown vis-studio 2005 yet)
Back to top
View user's profile Send private message
Jochen
KiXforms Devotee
KiXforms Devotee


Joined: 05 Mar 2003
Posts: 1204
Location: Stuttgart, Germany

PostPosted: Mon May 22, 2006 1:41 pm    Post subject: Reply with quote

Shawn wrote:
I dont have the full blown vis-studio 2005 yet


Which would cost how much ?

Maybe we could launch a donation wave with this thread Sceptic

_________________
Jochen

Tell me, and I will forget.
Show me, and I may remember.
Involve me, and I will understand.
Back to top
View user's profile Send private message MSN Messenger
Shawn
KiXforms Developer
KiXforms Developer


Joined: 22 Feb 2003
Posts: 1983
Location: Canada

PostPosted: Mon May 22, 2006 1:44 pm    Post subject: Reply with quote

Its not just that - I have to figure out what the impact of re-compiling kf.net under vis-studio 2005 and .net 2.0 really is. My big concern is whether things would still work under .net v1 - and my guess would be no.
Back to top
View user's profile Send private message
Jochen
KiXforms Devotee
KiXforms Devotee


Joined: 05 Mar 2003
Posts: 1204
Location: Stuttgart, Germany

PostPosted: Mon May 22, 2006 1:46 pm    Post subject: Reply with quote

Ouch ... so many scripts to break Head Hurts
Weeeell, better sooner than later then, no?

_________________
Jochen

Tell me, and I will forget.
Show me, and I may remember.
Involve me, and I will understand.
Back to top
View user's profile Send private message MSN Messenger
Display posts from previous:   
Post new topic   Reply to topic    KiXforms Forum Index -> KiXforms.NET Discussion All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group