FILE.TAIL([{lines}] would be good.

Give it a try, it costs you nothing !
m1zar00
Posts: 6
Joined: Thu Apr 22, 2021 5:27 pm
Has thanked: 4 times
Been thanked: 2 times

Re: FILE.TAIL([{lines}] would be good.

Post by m1zar00 »

Thanks all, for the great responses. I'm definitely going to look at the RTC storage, as that seems simplest to implement. And thanks for the code samples for scanning the file, interesting ideas!

Thank You!
User avatar
Fernando Perez
Posts: 106
Joined: Mon Feb 15, 2021 10:09 pm
Location: Santander (Spain)
Has thanked: 55 times
Been thanked: 54 times

Re: FILE.TAIL([{lines}] would be good.

Post by Fernando Perez »

Thanks to you and I assure you that Annex returns in satisfaction the time you dedicate.
But I reopen this topic because I need Cicciocb or someone who has used the "bas.rtcmem$" statement to clarify a very relevant question for me:
Does the data written to the RTC ram survive a complete system shutdown (power disconnection) or just a reset?
bugs
Posts: 46
Joined: Mon Feb 08, 2021 10:10 pm
Has thanked: 7 times
Been thanked: 20 times

Re: FILE.TAIL([{lines}] would be good.

Post by bugs »

Hi,

It is non-volatile (not RAM)
:oops:
User avatar
Fernando Perez
Posts: 106
Joined: Mon Feb 15, 2021 10:09 pm
Location: Santander (Spain)
Has thanked: 55 times
Been thanked: 54 times

Re: FILE.TAIL([{lines}] would be good.

Post by Fernando Perez »

I thought so, but I use this:

Code: [Local Link Removed for Guests]

bas.RtcMem$ = "Pepe y pepito"
a$ = bas.rtcmem$
wlog a$
And I save and execute the file.

Then I comment on the first line and repeat the save and run procedure.

Code: [Local Link Removed for Guests]

'bas.RtcMem$ = "Pepe y pepito"
a$ = bas.rtcmem$
wlog a$
If I do a reset, it works, but if I disconnect the ESP8266 and reconnect it, the Wifi connection breaks and wlog does not show anything.
bugs
Posts: 46
Joined: Mon Feb 08, 2021 10:10 pm
Has thanked: 7 times
Been thanked: 20 times

Re: FILE.TAIL([{lines}] would be good.

Post by bugs »

Ok - I think I had better keep quiet and wait for an expert to answer the question - sorry.
:?
cicciocb
Site Admin
Posts: 309
Joined: Mon Feb 03, 2020 1:15 pm
Location: Toulouse
Has thanked: 85 times
Been thanked: 197 times
Contact:

Re: FILE.TAIL([{lines}] would be good.

Post by cicciocb »

[Local Link Removed for Guests] wrote: [Local Link Removed for Guests]Tue Apr 27, 2021 5:47 pm Ok - I think I had better keep quiet and wait for an expert to answer the question - sorry.
:?
:D :lol:
Bugs don't worry, everyone makes mistakes the important is to recognise them ;)
Yes, this function survives at the reset but not at the power off.

However, for non-volatile information, the best is always to save into a file
m1zar00
Posts: 6
Joined: Thu Apr 22, 2021 5:27 pm
Has thanked: 4 times
Been thanked: 2 times

Re: FILE.TAIL([{lines}] would be good.

Post by m1zar00 »

Hi All,

Thanks again for all the ideas, and I've come up with one that seems pretty quick, regardless of file size or the need to pre-count lines.

Code: [Local Link Removed for Guests]

line$=""
fchr$=""
pos=FILE.SIZE(logfile$)-1
while fchr$<>chr$(13)
    pos=pos-1
    line$=fchr$+line$
    fchr$=FILE.READ$(logfile$,pos,1)
wend
wlog line$
So I basically work back from the file size until I find a carriage return (13), but could easily modify to be line-feed (10), and I have my last line. Cost in loops, is just the length of the last line. Appears to take around 87 milliseconds (Or 147 milliseconds on the test file Fernando Perez posted, his lines are longer...). I'm sure it hits the string handling a bit hard, as I'm adding the next character to the front of the string. I'm sure it's not optimal for long lines either. But I am now thinking of changing the log file to have a fixed length, then I can do this:

Code: [Local Link Removed for Guests]

line$ = FILE.READ$(logfile$,file.size(logfile$)-27,27)
Where 27 is the length of the string (plus the CR), which takes 48 milliseconds on my file (modified for a fixed line length).
Thanks all, for making me use my noodle. :D
User avatar
Fernando Perez
Posts: 106
Joined: Mon Feb 15, 2021 10:09 pm
Location: Santander (Spain)
Has thanked: 55 times
Been thanked: 54 times

Re: FILE.TAIL([{lines}] would be good.

Post by Fernando Perez »

I discard the use of bas.rtcmem$ for not surviving between full shutdowns and go back to the filesystem.
I have written this SUB:

Code: [Local Link Removed for Guests]

line$ = ""
t = 0
wlog "start..."

lastLine "/msft_250.csv", line$
wlog "250 lines processed in "; millis - t; " millliseconds"

lastLine "/msft_1000.csv", line$
wlog "1000 lines processed in "; millis - t; " millliseconds"

lastLine "/msft_3000.csv", line$
wlog "3000 lines processed in "; millis - t; " millliseconds"

lastLine "/msft_6000.csv", line$
wlog "6000 lines processed in "; millis - t; " millliseconds"

wlog chr$(13);"...finish"

END

SUB lastLine(fileName$, line$)
LOCAL a$, lastByte
  line$ = ""
  t = millis  ' for debug, delete in final version
  lastByte = file.Size(fileName$) - 1
  ioBuff.dim(0, 1)
  do
    file.read_ioBuff(0), filename$, lastByte, 1
    a$ = ioBuff.ToString$(0)
    line$ = a$ + line$
    lastByte = lastByte - 1 
  loop until a$ = chr$(10)
  ioBuff.destroy(0)
  wlog line$  ' for debug, delete in final version
END SUB
and I was surprised that in a short file it takes a few milliseconds, but (and I don't understand why) when I increase the number of rows in the file to 1,000, 3,000 and 6,000, the time increases to 2.5 seconds and stays that way regardless of the size of the file.
image.png
I will test your code to check speed in very large files
You do not have the required permissions to view the files attached to this post.
User avatar
Fernando Perez
Posts: 106
Joined: Mon Feb 15, 2021 10:09 pm
Location: Santander (Spain)
Has thanked: 55 times
Been thanked: 54 times

Re: FILE.TAIL([{lines}] would be good.

Post by Fernando Perez »

This is the result:

Code: [Local Link Removed for Guests]

line$ = ""
t = 0
wlog "start..."

logfile$ = "/msft_250.csv" : gosub lastline
wlog "250 lines processed in "; millis - t; " millliseconds"

logfile$ = "/msft_1000.csv" : gosub lastline
wlog "1000 lines processed in "; millis - t; " millliseconds"

logfile$ = "/msft_3000.csv" : gosub lastline
wlog "3000 lines processed in "; millis - t; " millliseconds"

logfile$ = "/msft_6000.csv" : gosub lastline
wlog "6000 lines processed in "; millis - t; " millliseconds"

wlog "...finish"

END

lastLine:
  t = millis ' for debug, delete in final version
  line$ = ""
  fchr$ = ""
  pos = FILE.SIZE(logfile$) - 1
  while fchr$ <> chr$(10)
    pos = pos - 1
    line$ = fchr$ + line$
    fchr$ = FILE.READ$(logfile$, pos, 1)
  wend
  wlog line$ ' for debug, delete in final version 
return
image.png
If you want my advice, you'd better use the variable length file line system, so the code will be reusable by other people and in other situations.
You do not have the required permissions to view the files attached to this post.
User avatar
Electroguard
Posts: 179
Joined: Mon Feb 08, 2021 6:22 pm
Has thanked: 40 times
Been thanked: 102 times

Re: FILE.TAIL([{lines}] would be good.

Post by Electroguard »

Touching on some earlier comments...
Non-volatile memory should be non-volatile, else it defeats the purpose of being non-volatile memory.
From my own very volatile biological memory of more than a year ago I seem to remember that RTCMEM$ did actually retain contents through a power-up, but that they got jumbled up, which suggested that it was actually non-volatile but that there was a code hiccup in the handling of the memory, perhaps inherited from arduino.
I know I had to abandon a useful snippet for saving variables such as reboots=nn (rolling count of the number of device reboots) into RTCMEM$ using word.setparam$ and word.getparam$, but it was easy enough just to do the same to a file instead of the unreliable volatile memory, so was more of a disappointment than a problem.

rtcmem$ = bas.rtcmem$
restarts$ = word.getparam$(rtcmem$,"Restarts")
restarts$ = str$(val(restarts$) + 1)
word.setparam rtcmem$, "Restarts", restarts$
bas.rtcmem$ = rtcmem$
...

SAVE:
word.setparam rtcmem$, "Script", bas.filename$
word.setparam rtcmem$, "Breadcrumb", breadcrumb$
word.setparam rtcmem$, "Time", time$
word.setparam rtcmem$, "Date", date$
word.setparam rtcmem$, "Ramfree", str$(ramfree)
word.setparam rtcmem$, "Flashfree", str$(flashfree)
word.setparam rtcmem$, "Restarts", restarts$
word.setparam rtcmem$, "Cause", cause$
' word.setparam rtcmem$, "Err num", str$(bas.errnum)
' word.setparam rtcmem$, "Err msg", bas.errmsg$
' word.setparam rtcmem$, "Err line", str$(bas.errline)
bas.rtcmem$ = rtcmem$
return
Post Reply