Manging data between 2 files

1

Now that I got to Identify data in TMemoryStream I can see my data correctly.
thanks Paul_McGee
My focus is building many sub files to be stored in a single file and not be bound by the file system text parameters

I have a main file with my TFileStream object and a 2nd identical file that opens a new file automatically and simplifies getting and saving data out of the file to TMemoryStreams and that seems to work well under windows.

I am taking the main file with my TFilestream to copy its data to a 2nd file and installing the modifrcations
I have copyied the updated data back to the main file and then reused and it works good but this is extra deamanding on the hard drive that I pefer not to do.
No matter what I do - swaping files names on the files by first closing my TFilestream’s and renaming the files and then reopening my TFilestreams. And the idea to change handles in my TFilestreams. none of this works.
Its my lack of experience with programiclly handling files and knowing the best way to do this process as I like my TFilestream objects as they are doing the simplication to the saving and loading of file data that I do not prefer to use low level file instructions.

This may sound crazy where I have no idea if anything like this can work but windows has something with pictures or meta files that store data partly in RAM and when over sized automatically uses the hard drive. To use that service can offer speed and many be simplicity to access the data or their many be another tool in windows to use.

Has any one has experience with using files like this

2

I’m struggling to make sense of what you are doing or want to do.

Please take a step back and write down the abstract specification of what it is you are planning to build, to serve as a starting point for a fruitful discussion on implementation. :slight_smile:

3

So I’m storing many sub files in a single file and so I’m rebuilding to a 2nd file with the changes as sub files get changed in the main file.
But I need system to move the data from the 2nd file to make it the Main file when I have finished rebuilding the new file.
Releasing the windows handles on the files and changing the names before reassessing the files does not like to work for me
I want some idea’s of what others would do in this situation

4

Ok. I think many people probably wouldn’t approach this this way … BUT … leaving that aside …

What is the trigger for bringing the Maxxy-File in from disk, to memory ?

What are the things that cause some Sub-Files to be modified?

And what triggers the writing of the Maxxy-File back to disk?

I can’t see any reason to specifically keep separate files ACTUALLY contiguous while they are in memory.
They aren’t going to take up more space, just because they are logically separate .. and they will be MUCH easier to deal with.

Let’s say you have a record TMemSubFile, and an array of these records :

type
   TMemSubFile = record
     Filename  : string;
     SubFileMS : TMemoryStream;
   end;

   TMemSubFileArray = TArray<TMemSubFile>;

Then the very first time, for each (sub-)file, you can add a record to the array and read in the data.

You can then change each file independently; Delete them, re-order them, add new ones, whatever …
And then finally when you’re done, create a single TFileStream and append them all one by one, as you write it out to disk.

You could keep a directory of information regarding how many files, and what file starts at what byte number, at the start of the stream. Or at the end of it maybe. Or just keep it separately.

Then when you bring it all back into memory next time you read in the directory info, open the Maxxy-File, and recreate all the Sub-Files again in memory.

5

Ok, that sounds like it should be doable with simple file operations, how exactly are you opening the main file in the first place?

Conceptually, its something like

  • close both files
  • rename main file to temp file name
  • rename 2nd file to become the new main
  • if successful rename, delete temp file
  • reopen main
6

I’m also having trouble understanding what’s trying to be developed. If I’m processing a complex task and want to save the results in a manner that can be used between two Delphi apps (or loaded back into memory at a later point in time) then i save the data into a record. The record can contain complex data structures and dynamic arrays. I then use the ukbdynamic GitHub - KrystianBigaj/kblib: Automatically exported from code.google.com/p/kblib to then save the record to the hard drive which can be loaded at a later point in time. ukbdynamic can handle very large memorystreams but you need the identical record structure and identical bitness in both apps.

7

Because I’m making my own version of PKzip that is not file name style dependant to contain memory streams of different sizes. What I did that seems to work.
Is I gave my self - two TFileStream and I gave extra pointers to the File Streams - old File Stream and New File Stream. Then I can swap them around when the new one got up dated and was then made into old one.
I made two memory streams to contain my sub file control data on each file and as sub files grew so the Memory streams grew as this is stored at the top of the file and I rebuilt the new file after the control data as it got moved over. Some one may rather use arrays may be.

The closing and opening renaming of files and reopening is not working for some reason. It could be a bug in my code as its new.

I found in the end this code below was the best way to look into a memory streams block data for debug/tracing purposes in the end.

Type
  AArray = array[0..100] of Char;
  PA = ^AArray;
var
  PAR: PA;

Placed PAR in my code to come up in my debug/tracing

Debugging services in Delphi with streams is not always the best - The issue is a memory stream has object data and a block of data where tracing you need to see both types or at least the start of the block of data and parameters like data size. The Delphi 7 IDE showed better debug information the Delphi 12 its for ever putting in “End” into my text that I spend too much time removing ‘end’ and the stupid endless pup ups telling me nothing but only covering my is not good. and you try to turn the IDE settings off and it resets its self all the time.

This is where I’m up to and making or adding to the file I have up and working - so that is the main stuff. I have to delate and resize yet and bugs

8

Hi Lex,

It is totally possible to use one TFileStream (TBufferedFileStream would be better if you are reading/writing in small chunks) to read the existing file and another TFileStream to write the modified file to a different filename and then free the streams (release any file handles), then rename the original file to .bak (as a precaution), rename the second file to the first filename, then delete the original.bak file.

Alternatively, if you can always fit two copies of the file file into memory (if they are not huge files) then you can load the entire original file into one TMemoryStream and write the modified file to a second TMemoryStream then save that to the existing filename. You could even use a TFileStream for the second one to write the new file directly without needing a second copy in memory but that may be subject to file corruption unless your file updating code is very robust.

As for debug view, as I mentioned in the other thread since you are using Delphi 12, the built in debug visualizer can display the contents of the TMemoryStream as hex bytes or text.

Cheers,
Jarrod

9

Jarrod Hollingworth - TBufferedFileStream that is a good idea to increase file access speed and buffers the saving of data - very good idea

Paul_McGee creating a directory and many separate files I totally see the point of that as file size slows things down to access through a file’s data. For me that is a total rebuild at this point. This option to me means system control data with every created file and enables many safeties like a system crash rebuild - so many things to consider

building from one file to the next means does not leave much room for backs ups?

10

TBufferedFileStream.
FlushBuffer I take this writes the complete file to the disk drive
my code at the moment is constantly setting the Position and doing reads or saves
Read(Strm.Memory^, Strm.Size);
Write(Strm.Memory^, Strm.Size);
With TFileStream and TMemoryStream and most of the time reads and writes most of the file from one end to the other with little interruptions.
I Have the options below where no memory stream is planed to be over a DWord in size
so with the below methods what is the correct methods to use WriteBuffer or WriteBufferData to get the buffering as my data can have a wide venation in size as I expect ordinary compressed picture storage is somewhat common.
I don’t know what size is the common buffering Size when reading data

Write
Write64
WriteBuffer
WriteBufferData

Read
Read64
ReadBuffer
ReadBufferData

11

That is interesting, 'cos I was thinking that the one reason I could imagine to have to keep everything contiguous would be if the entire stream was an encryption and every byte depended on previous bytes.
But even then, I /think/ that programs like PkZip etc tend to compress on a file by file basis …

12

Compression can be nice but I do not think it works with pictures so its not much of a saving.
Encryption I can add my self as its not hard code to make.
Military encryption has to work at much higher levels to domestic methods
Encryption has upgraded a long way from every byte depended on previous bytes as that is as old has world war 2.
The best thing with encryption is do not use documented encryption as US police has full access to all that. I read once a south African drug lord made his own encryption code and police could not break the encryption - they caught him through cell phones instead.
But how much work do you want into your own encryption if you study the subject that it gets to the point even AI can read 1,000 different sentences if you want.
You can even add extra text in a set of rules made with your code as well.

13

No? Look it up in the help instead of assuming? :slight_smile: