KiXforms Forum Index KiXforms
The Forum for the KiXforms Community
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 
 Quick Links 
Site News
Downloads
Documentation
Donations
Script Archive
Tracking Systems

kEnc - bin2str encoding engine for kixtart

 
Post new topic   Reply to topic    KiXforms Forum Index -> Discussion
View previous topic :: View next topic  
Author Message
Lonkero
KiXforms Devotee
KiXforms Devotee


Joined: 13 Mar 2003
Posts: 1022
Location: Espoo, Finland

PostPosted: Thu Jan 19, 2006 2:40 pm    Post subject: kEnc - bin2str encoding engine for kixtart Reply with quote

some of have used base64 already with our kixforms scripts and noticed that it is a sad case which adds about 30-40% to the original size.

I've been thinking for kixtart compression for a while and even done some encoding and compression engines but they are way too slow with current kixtart string like parsing.
as example, I wrote a yEnc engine that was compatible with kixtart and after about half'n hour later I had to kill it.
it just was too slow.
anyway, the size positive was so good (about 2% of increase in size) and the technique so simple that I decided to write a compiler version out of it.

result is kEnc.
one can't do much of it now.
just convert binaries (I used wkix32.exe in all my testing) to string, add to your script and then decode it.

but to really be usable, shawn would need to add a fromKEncString() to kixforms.

now, as the usability is low, I place it online so ppl interested in new stuff can play with it.



kEnc.zip
 Description:
kEnc - version 1

Download
 Filename:  kEnc.zip
 Filesize:  24.9 KB
 Downloaded:  959 Time(s)


_________________
Hammer
Back to top
View user's profile Send private message Visit poster's website MSN Messenger
Shawn
KiXforms Developer
KiXforms Developer


Joined: 22 Feb 2003
Posts: 1983
Location: Canada

PostPosted: Thu Jan 19, 2006 3:06 pm    Post subject: Reply with quote

wow, this is very cool. I encrypted a jpg and it only increased about 5% in size. Do you see any issues with copy/pasting the results into a script, and in Kixtarts ability to read it without errors, have you tried it yet ?
Back to top
View user's profile Send private message
Lonkero
KiXforms Devotee
KiXforms Devotee


Joined: 13 Mar 2003
Posts: 1022
Location: Espoo, Finland

PostPosted: Thu Jan 19, 2006 8:41 pm    Post subject: Reply with quote

no, didn't try it but that's the whole idea behind it Smile
I scanned for character codes with which kixtart has issues and think I got them all.
but, will see only after one tries.

_________________
Hammer
Back to top
View user's profile Send private message Visit poster's website MSN Messenger
Lonkero
KiXforms Devotee
KiXforms Devotee


Joined: 13 Mar 2003
Posts: 1022
Location: Espoo, Finland

PostPosted: Thu Jan 19, 2006 9:56 pm    Post subject: Reply with quote

    Test 1
    - read encoded wkix32.exe with readline() and write it to another file with writeline(). decode it. test functioning by running a script with it.
    result: passed.
    additional notes. kixtart writes the data "wrong". for some odd reason, additional line is added after each one when using @crlf.

    Test 2
    -embed the string as variable and write it back to a file.
    result: failed.
    there is still some character I failed to see which kixtart can't handle.

_________________
Hammer
Back to top
View user's profile Send private message Visit poster's website MSN Messenger
Lonkero
KiXforms Devotee
KiXforms Devotee


Joined: 13 Mar 2003
Posts: 1022
Location: Espoo, Finland

PostPosted: Fri Jan 20, 2006 12:18 am    Post subject: Reply with quote

used this code to find the problematic values...
well...
all the ones that failed are already escaped so dunno.
maybe I used too large string (a bit over 250 000 characters)
Code:

$=setoption("novarsinstrings","on")
del ".\critic_print.txt"
$= open(1,".\critic_print.txt",5)
for $=1 to 255
if $<>26 and $<>39
 $s = writeline(1,"$c"+$+" = '"+chr($)+"'"+@crlf)
endif
next
$= close(1)
call ".\critic_print.txt"

for $x=1 to 255
 $s = execute("? "+$x+" ' ' asc($c"+$x+") '   '")
 $s = execute("if "+$x+"<>asc($c"+$x+") '   <===' endif")
next

get $

_________________
Hammer
Back to top
View user's profile Send private message Visit poster's website MSN Messenger
Lonkero
KiXforms Devotee
KiXforms Devotee


Joined: 13 Mar 2003
Posts: 1022
Location: Espoo, Finland

PostPosted: Fri Jan 20, 2006 12:31 am    Post subject: Reply with quote

k, the problem found.
string size limitation.
after dividing the data to multiple vars and using them, worked fine.
question is, should the encoder add a extra empty file after each 32k of chars to ease the division job?
or add a chunk(split) option?

_________________
Hammer
Back to top
View user's profile Send private message Visit poster's website MSN Messenger
Lonkero
KiXforms Devotee
KiXforms Devotee


Joined: 13 Mar 2003
Posts: 1022
Location: Espoo, Finland

PostPosted: Fri Jan 20, 2006 12:32 am    Post subject: Reply with quote

oh.
shawn, did you change the offSet value from the default?
got the smallest result with wkix32.exe with offset 90

_________________
Hammer
Back to top
View user's profile Send private message Visit poster's website MSN Messenger
Lonkero
KiXforms Devotee
KiXforms Devotee


Joined: 13 Mar 2003
Posts: 1022
Location: Espoo, Finland

PostPosted: Fri Jan 20, 2006 5:50 pm    Post subject: Reply with quote

k, tested again.
yes, it is the string limitation thingie.
there is no limitation for strings but only inside the kixtart script.
that is, if one wants to use longer strings in variable values set in the script (not loaded from anywhere and not set during execution) one needs to split the string to some chunks.
or use "" + "" method thus chunking it.

_________________
Hammer
Back to top
View user's profile Send private message Visit poster's website MSN Messenger
Lonkero
KiXforms Devotee
KiXforms Devotee


Joined: 13 Mar 2003
Posts: 1022
Location: Espoo, Finland

PostPosted: Sat Jan 21, 2006 11:19 am    Post subject: Reply with quote

and the final escape character list is as follows:
Code:

const char KIXCRITICALS[] = {1, 10, 13, 26, 34, 39, 64, 36, 37, 0};

_________________
Hammer
Back to top
View user's profile Send private message Visit poster's website MSN Messenger
Lonkero
KiXforms Devotee
KiXforms Devotee


Joined: 13 Mar 2003
Posts: 1022
Location: Espoo, Finland

PostPosted: Sat Jan 21, 2006 12:44 pm    Post subject: one more test... Reply with quote

we should make a test how well binary data is transferred over the net like this.
as example, FTP clients with text mode set to on should have troubles as they screw up (at least should) the highbits of the data.

so, if any user around here is kind enough and logs in with their MSN or GTalk clients and helps me try this out... I'd be glad Smile

_________________
Hammer
Back to top
View user's profile Send private message Visit poster's website MSN Messenger
Lonkero
KiXforms Devotee
KiXforms Devotee


Joined: 13 Mar 2003
Posts: 1022
Location: Espoo, Finland

PostPosted: Sun Jan 22, 2006 12:20 am    Post subject: Reply with quote

hmm...
think I will add some compression to this enc system.

_________________
Hammer
Back to top
View user's profile Send private message Visit poster's website MSN Messenger
Lonkero
KiXforms Devotee
KiXforms Devotee


Joined: 13 Mar 2003
Posts: 1022
Location: Espoo, Finland

PostPosted: Mon Jan 23, 2006 5:00 pm    Post subject: Reply with quote

new version.
instead of printing huge amount of same characters after each other, this one truncates them and regenerates during output = compression.

the level of compression is about the same as the overheat.
thus, wkix32.exe compressed to kEnc string of just some hundred bytes smaller.

again, not sure if this is the best approach or should I go with higher level.
on the other hand, this is not a compressor.

anyways, here it is for anyone (guess that's only shawn) to try.



kEnc.zip
 Description:
kEnc - bin2str encoder for KiXtart by your's truly, 2006

Download
 Filename:  kEnc.zip
 Filesize:  25.09 KB
 Downloaded:  1121 Time(s)


_________________
Hammer
Back to top
View user's profile Send private message Visit poster's website MSN Messenger
Lonkero
KiXforms Devotee
KiXforms Devotee


Joined: 13 Mar 2003
Posts: 1022
Location: Espoo, Finland

PostPosted: Mon Jan 23, 2006 5:05 pm    Post subject: Reply with quote

oh, and the files produced by the different versions are fully compatible. (have you ever herd that with beta products? Wink)

also, did a quickie test. set the offset to 90 and resulting filesize for wkix32.exe:
252 237 Bytes

when original was:
253 952 Bytes

that makes compression of over 1kB! Smile
so, overheat of produced by kEnc is -1% - 5% Very Happy

_________________
Hammer
Back to top
View user's profile Send private message Visit poster's website MSN Messenger
Display posts from previous:   
Post new topic   Reply to topic    KiXforms Forum Index -> Discussion All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group