FILE.TAIL([{lines}] would be good.

Give it a try, it costs you nothing !
m1zar00
Posts: 9
Joined: Thu Apr 22, 2021 5:27 pm
Has thanked: 9 times
Been thanked: 2 times

FILE.TAIL([{lines}] would be good.

Post by m1zar00 »

I can't see an easy way of doing this currently, please let me know if I missed something. I need to load the last line of a data log file (CSV) when Annex starts up, however there doesn't appear to be a FILE option for that. I would use the FILE.READ(line... style, but I don't know the last lines number. It would be great if I could have something like FILE.TAIL([{lines}]) where it returns the last line of a file or 'lines' of a file. It's not essential, but is the way I was used to initialising the internals of my script with the last known (and probably closest) readings. I don't really want to script something to read the whole log file, just to get the last lines. I'm sure that could be done, but would be a lot faster if done internally.
Thanks,
James
bugs
Posts: 142
Joined: Mon Feb 08, 2021 10:10 pm
Location: Scotland
Has thanked: 44 times
Been thanked: 50 times

Re: FILE.TAIL([{lines}] would be good.

Post by bugs »

Hi,

No - you have not missed anything.
Don't bother trying to read a large file line by line as it takes a g e s...

Assuming you are appending data to the csv file then simply either write the latest data (only) to another file or, if <512 bytes, write it to RTC RAM (BAS.RTCMEM$) and read it back instantly next time Annex starts up.
User avatar
cicciocb
Site Admin
Posts: 1899
Joined: Mon Feb 03, 2020 1:15 pm
Location: Toulouse
Has thanked: 407 times
Been thanked: 1269 times
Contact:

Re: FILE.TAIL([{lines}] would be good.

Post by cicciocb »

Hi,
I'll try to see what can be done.
m1zar00
Posts: 9
Joined: Thu Apr 22, 2021 5:27 pm
Has thanked: 9 times
Been thanked: 2 times

Re: FILE.TAIL([{lines}] would be good.

Post by m1zar00 »

Hi,

@bugs, Thanks, I'll look at your suggestions, I like the RTC RAM idea, but I'm not using one at the moment. I'll look into writing the last line to both the log file and another as a work-around. I was also thinking that I could get the log file size, divide it by the average line length (it varies by a few bytes so will under-shoot it), and then read lines from the calculated line number onward. As long as I under shoot the line I want, it might speed up getting to the last line. The file is going to grow to about 350k-400k, so that's why I'm trying to avoid reading all the lines.

@cicciob, Firstly, Annex RDS is really great, I'm having a lot of fun with it. Having all the libraries for the devices already loaded makes it tremendously powerful for my kind of projects, so thank you. If you're able to add a feature that would be great. Since you already offer reading from a particular line, perhaps just a count of lines would then allow me to read the last one using the current read line.

Thank You!
User avatar
Electroguard
Posts: 836
Joined: Mon Feb 08, 2021 6:22 pm
Has thanked: 268 times
Been thanked: 317 times

Re: FILE.TAIL([{lines}] would be good.

Post by Electroguard »

I like the RTC RAM idea, but I'm not using one at the moment.
Just to clear up any confusion...
Despite the name similarity, an RTC (real time clock) module is optional, whereas RTC RAM is an area of non-volatile ESP memory which anyone can access with the Annex command:
bas.rtcmem$
(This memory is limited to 512 bytes and can be set with the corresponding command BAS.RCTMEM$ = “xxx”)

So if wanted, you do actually have 512 bytes of non-volatile memory available to use for storing any info that you may want to survive a reboot.
User avatar
cicciocb
Site Admin
Posts: 1899
Joined: Mon Feb 03, 2020 1:15 pm
Location: Toulouse
Has thanked: 407 times
Been thanked: 1269 times
Contact:

Re: FILE.TAIL([{lines}] would be good.

Post by cicciocb »

HI,
I also think that the to use the INTERNAL RTC memory as suggested by Bugs and Electroguard is the easiest solution.
I understand that you want just check last record and add more data only if it has changed.

Take into account that reading the file line by line is very slow as the file is open and closed at each call.
This function was done considering little files (few KB) and is not adapted to big files.
Counting the number of lines can be done but I'll see how give more flexibility on the file functions.

If someone has some ideas, you are welcome.
User avatar
Electroguard
Posts: 836
Joined: Mon Feb 08, 2021 6:22 pm
Has thanked: 268 times
Been thanked: 317 times

Re: FILE.TAIL([{lines}] would be good.

Post by Electroguard »

A couple of things you could try...

Keep a running total of how many LINES you write to the LOG, and save bas.rtcmem$=LINES each time of writing to the log, thereby allowing you to specify the last line number of the log at any time for reading back.

Or sequence through the log file reading a line at a time and discarding the previous until reaching the end of the log file (eg: an empty line).
If you counted the LINES as you step through, then even if it takes too long to do at startup each time, at least it would give you a tool to discover the number of lines in the current log file to save using bas.rtcmem$=LINES
User avatar
Fernando Perez
Posts: 378
Joined: Mon Feb 15, 2021 10:09 pm
Location: Santander (Spain)
Has thanked: 195 times
Been thanked: 267 times

Re: FILE.TAIL([{lines}] would be good.

Post by Fernando Perez »

I agree with you that the best solution is to use bas.rtcmem$, but if your file is not very large and the boot time of your system is not too important, you could use something similar to this:

Code: [Local Link Removed for Guests]

nLines = 0
fileName$ = "/msft.csv"
t = millis
wlog "start..."
gosub tail
wlog nLines; " in "; millis - t; " millliseconds"
END

tail:
  ioBuff.dim(0, file.Size(fileName$))
  file.read_ioBuff(0), filename$
  for i = 0 to ioBuff.len(0)
    if ioBuff.toHex$(0, i, 1) = "0A" then nLines = nLines + 1
  next i
  ioBuff.destroy(0)
return
I am attaching a csv file of 16.633 bytes and 238 lines to experiment. It takes 8 and a half seconds.
https://myrapidq.it/public/msft.csv

By the way, Cicciocb, I have verified that if I omit the size parameter in the IObuff.ToHex $ (buff_num, [, start [, size]]) the memory overflows.
bugs
Posts: 142
Joined: Mon Feb 08, 2021 10:10 pm
Location: Scotland
Has thanked: 44 times
Been thanked: 50 times

Re: FILE.TAIL([{lines}] would be good.

Post by bugs »

Sorry - my fault for introducing the RAM confusion.
I blame "memory leak" from a previous AVR project using a DSxxxx RTC chip and its RAM.

Maybe a file.linecount function would be useful to know which line to read.
User avatar
Fernando Perez
Posts: 378
Joined: Mon Feb 15, 2021 10:09 pm
Location: Santander (Spain)
Has thanked: 195 times
Been thanked: 267 times

Re: FILE.TAIL([{lines}] would be good.

Post by Fernando Perez »

The response time can be improved by this other procedure, but they are still excessive for files of 400 Kbytes.

Code: [Local Link Removed for Guests]

wlog "start..."
nLines = 0
fileName$ = "/msft.csv"
t = millis
gosub tail
wlog nLines; " in "; millis - t; " millliseconds"
wlog file.read$(fileName$, nLines)
END

tail:
  line$ = file.read$(fileName$, 2)
  nLines = int(file.Size(fileName$) / len(line$))

  do
    nlines = nLines - 1
  loop until file.read$(fileName$, nLines) <> "_EOF_"

return
If nobody proposes something better, I also think that the most suitable solution is to save the number of the last written line in RAM RTC.
Post Reply