Input Output Files Format

Input Output Files Format• At their lowest level, all files written to your

computer's hard drive are a series of binary bits.

• However, in LabVIEW, four of the most common techniques for storing data are the ASCII file format, direct binary storage, TDM file format and LVM file formats.

• Each of these formats has advantages and some formats work better for storing certain data types than others.

ASCII File Format

Text format files store data using the ASCII character set. To store data in text format, use the String functions to convert all data to text strings. Text files can contain information of different data types stored as strings. Use text format files when:

ASCII File Format• Text files typically take up more memory than binary and

datalog files if the data is not originally in text form, such as graph or chart data, because the ASCII representation of data is usually larger than the data itself.

• Additionally, it is difficult to randomly access numeric data intext files. Although each character in a string takes up exactly1 byte of space, the space required to express a number as text typically is not fixed. You might lose precision if you store numeric data in text files.

Binary Files• Use binary files to save numeric data and to access specific

numbers from a file or randomly access numbers from a file.

• Binary files are machine readable only, unlike text files, whichare human readable.

• They are the most compact and fastest format for storing data.

• Binary files are more efficient because they use less disk spaceand because you do not need to convert data to and from a text representation when you store and retrieve data.

• Storing binary data, such as an integer, uses a fixed number of bytes on disk.

Binary Files• A binary file can represent 256 values in 1 byte of

disk space. • Often, binary files contain a byte-for-byte image of

the data as it was stored in memory, except for cases like extended and complex numeric values.

• When the file contains a byte-for-byte image of the data, as it was stored in memory, reading the file is faster because conversion is not necessary.

• For example, storing any number from 0 to 4 billion in binary format, such as 1, 1,000, or 1,000,000, takes up 4 bytes for each number.

Binary Files• The following are some of the common functions that interact

with binary files.

Binary Files

Writing to Binary Files

• Following is an example that writes an array of doubles to a binary file. Refer to the Arrays section for more information about the Prepend array or string size? option.

Boolean values Representation• LabVIEW represents Boolean values as 8-bit values

in a binary file. • A value of all zeroes represents False. • Any other value represents True. • This divides files into byte-sized chunks and

simplifies reading and processing files.• To efficiently store Boolean values, convert a series

of Boolean values into an integer using the Boolean Array To Number function. Two methods for writing six Boolean values to a binary file are displayed.

Boolean values Representation

8-bit integers Representation• Unsigned 8-bit integers (U8s) directly correspond to

bytes written to the file. • When you must write values of various types to a

binary file, convert each type into an array of U8s using the Boolean Array To Number, String to Byte Array, Split Number, and Type Cast functions. Then, you can concatenate the various arrays of U8s and write the resulting array to a file.

• This process is unnecessary when you write a binary file that contains only one type of data.

8-bit integers Representation

Strings Representation• Strings are stored as a series of unsigned 8-bit integers, each of

which is a value in the ASCII Character Code Equivalents Table.

• This means that there is no difference between writing strings with the Binary File Functions and writing them with the Text File Functions.

Arrays Representation• Arrays are represented as a sequential list of each of their

elements. • The actual representation of each element depends upon the

element type. • When you store an array to a file, you have the option of

preceding the array with a header. • Following is an example of writing a 2-dimensional array of 8-

bit integers to a file with a header. • The prepend array or string size? terminal of the Write Binary

File function enables the header. • Notice that the default value of this terminal is True.

Therefore, headers are added to all binary files by default.

Arrays Representation

Reading from Binary Files• When reading a binary file, there are two methods of accessing

data.• The first is to read each item in order, starting at the beginning

of a file. This is called sequential access and works similar toreading an ASCII file.

• The second is to access data at an arbitrary point within the file. For example, if you know that a binary file contains a 1-D array of 32-bit integers that was written with a header and you want to access the tenth item in the array, you could calculate the offset in bytes of that element in the file and then read only that element. In this example, the element has an offset of 4 (the header) + 10 (the array index) * 4 (the number of bytes in an I32) = 44. Accessing data in this way is known as random access.

Sequential Access• To sequentially access all of the data in a file, you can call the Get

File Size function and use the result to calculate the number of items in the file, based upon the size of each item and the layout of the file.

• You can then wire the number of items to the count terminal of the Read Binary function.

• Each read operation updates the position within the file so that you read a new item each time the read function is called.

• When using this technique to access data you can check for the End of File error after calling Read Binary or calculate the number of reads necessary to reach the end of the file by using Get File Size.

Sequential Access

Random Access

• To randomly access a binary file, use the Set Position VI to set the read offset to the point in the file you want to begin reading.

• Notice that the offset is in bytes. Therefore, you must calculate the offset based upon the layout of the file. In the following example, the VI returns the array item with the index specified, assuming that the file was written as a binary array of double precision numerics with no header.

Random Access

Datalog Files• A specific type of binary file, a datalog file is the

easiest method for logging cluster data to file. • Datalog files store arrays of clusters in a binary

representation. • Datalog files provide efficient storage and random

access, however, the storage format for datalog files is complex, and therefore they are difficult to access in any environment except LabVIEW.

• In order to access the contents of a datalog file, you must know the contents of the cluster type stored in the file.

Datalog Files• Datalog files store a list of records to a file.• Each record is represented by a cluster, and can contain

multiple pieces of data of any data type. Datalog VIs allow you to read and write arrays of clusters to and from datalogfiles.

• When you open a datalog file, you must specify the record type used by the file.

• To do this, wire a cluster of the appropriate type to the Open/Create/Replace Datalog function.

• After the file is open, you program datalog files like any other binary file.

• Random access is available, although offsets are specified in records instead of bytes.

Datalog Files

Datalog Files

• The following figure depicts an example of randomly accessing a datalog file by setting the read position with the Set Datalog Position function.

• To sequentially access data in a datalog file, wire the output of the Get Number of Records function to the count terminal of the Read Datalog function.

Datalog Files

TDM Files• Test Data Exchange Format (TDM) is a hybrid file format that

combines Binary storage and XML formatted ASCII data. • In a TDM file, the raw numerical data is stored in binary

format. This provides the advantages of binary, such as efficient space usage and fast write times.

• In addition to this data, an XML format stores the structure ofthe data and information about the data.

• This allows the information in the file to be easily accessible and searchable. Typically, the binary data and XML data are separated into two files, a .tdm file for the XML data and a .tdx file for the binary data.

TDM Files

TDM Files• The following are benefits of using TDM

files.

TDM Files

• TDM files each contain a File object and can contain as many Channel Group and Channel objects as you want. Each of the objects in a file has properties associated with it, which creates three levels of properties that you can use to store data. For example, test conditions are stored at the file level, and UUT information is stored at the channel or channel group level.

TDM Files

Methods for Creating TDM Files

• In LabVIEW, you can create TDM files in two ways.

Using the Express VIs• The Write to Measurement File and Read from Measurement

File Express VIs allow you to quickly save and retrieve data from the TDM format.

• The configuration dialog box for the Write to Measurement File Express VI is displayed.

• Notice that you can choose to create a LVM or a TDM file type.

• However, these Express VIs give you little control over your data grouping and properties and do not allow you to use the features that make TDM files useful, such as searching for data based upon conditions.

Using the Data Storage API VIs• To gain access to the full capabilities of TDM files,

use the Data Storage API.

• The Data Storage API is a set of VIs that can write multiple file formats, however, they write TDM files by default.

• An example of a simple program that logs channel properties and numeric data to a TDM File using the Data Storage API is displayed

Using the Data Storage API VIs

Data Storage APIThe Open Data Storage VI opens a reference to a TDM file. You can hard code a file path by using the configuration dialog or determine the path at runtime by using the block diagram terminal. The Configure Open Data Storage dialog box is displayed

Data Storage API• The Write Data VI allows you to create channels

and channel groups within your file. It also allows you to write properties and data for the item you create. The configuration dialog for this VIallows you to select which properties have block diagram terminals and specify how the VI behaves if you attempt to store two channels in a file with the same name.

Data Storage API• The Read Data VI allows you to search for channels or channel groups

based upon conditions you specify. • The configuration dialog for this VI allows you to specify the conditions

for the search, as well as the type of data returned when the search result is a channel. This VI can return the actual data signal from a channel, but in order to access other properties of a channel or channel group, pass the references returned from this VI to the Get Properties VI.

• Notice that because a query can have multiple results, this function returns all of its results, including refnums, in arrays

Data Storage API• The Set Properties VI allows you

to set properties on a channel, channelgroup, or file. Because the Write DataVI allows you to set the properties of achannel or channel group, this VI is most often used to set file properties.

• The configuration dialog allows you to select the type of object to set properties for, and the properties to set. You canalso use this VI to set the data signal of a

channel.

Data Storage API• The Close Data Storage VI closes a reference to a TDM File.

Notice that you only must close the file reference; any references that you acquire to channels and channel groups close automatically when you close the file reference.

Constructing Queries• Constructing queries helps read data for viewing or analysis.

Perform basic queries using only the Read Data VI. A simple query that graphs the signal data from all Channels in the file with a maximum greater than or equal to 5 is displayed.

Constructing Queries• To access properties other than the data signal, use the Get Properties VI

with the result references from Read Data. Notice that because Read Data returns an array, you must use a For Loop to index it before calling Get Properties. Rather than displaying the signal data, the following example displays channel properties. The example returns the Name, Description, Minimum and Maximum of all channels with a Maximum greater than or equal to 5.

Constructing Queries • When your TDM file contains channel groups, you often want to search for

channels only in a particular group. You can do this by using two Read Data VIs, one to search for the appropriate channel group and a second to search for channels within that group. Wiring the reference of the channel group to the Read Data VI allows you to constrain the channel search to

channels within that group.

Constructing Queries• To construct complex TDM queries, use the Merge Queries VI. This VI

allows you to combine the results of two Read Data VIs. When you use Merge Queries, you can return results that are in the first query and the second query or combine the results of both queries. After calling Merge Queries, you often call Get Properties to access the signals or properties of the query results.

Grouping Data• Carefully consider the best way to group your data as the data grouping can

have a significant impact on both the execution speed and implementation complexity of writes and queries. Consider the original format of your data and how you want to search or view the data when choosing a grouping scheme. One grouping technique is to group data by the type of data.

TDMS Files• TDMS (Technical Data Management - Streaming)

which is are binary files in an internal LabVIEW file format. The TDMS format gives more effective and accurate data storage than the LVM format. TDMS files may be opened in LabVIEW, of course, and in NI DIAdem which is a software tool for formanaging, analyzing, and reporting data in logfiles. TDMS files can save data in an organized way using a number of Groups and a number of Channels within a given Group. (The older TDM file format is still supported.)

Writing data to a TDMS file

Writing data to a TDMS file• The TDSM Write function writes the data continuously to the file. (It

contains a data buffer so that the While loop is not unnecessarily delayed by the file writing.)

• The Case structure is used to start, and stop, the file writing. (In the figure the True case is shown. The False case does not contain any code except the file path wire from the input tunnel to the output tunnel.)

• Three signals - t, u, y - are collected with a Merge Signals function (which is on the Express / Signal Manipulation palette) before they are fed into the the TDMS Write function.

• The channel group name and the channel names are written to the logfiletogether with the data. This is convenient for later use of the data, e.g. when opening the data in the DIAdem tool.

• In this example the points of time generated by the Elapsed Time function are saved as data together with the values of u and y. (If the data to be saved are Waveform data the time information is contained in the data, and then it is not necessary to define the time signal explicitely as in the example above.)

Reading data from a TDMS file

Reading data from a TDMS file• If you run the prevoius VI the TDMS Viewer is also opened, in addition to

the Front panel, see the figure below. The TDMS Viewer is a graph tool for quickly plotting the channels in the logfile. Note that you have to click the Quit button in this window to quit the TDMS Viewer, and to stop the VI.

Reading data from a TDMS file

• The VI does not contain any While loop since the read from file operation is done once. (This is the typical situation.)

• The TDMS Read function (on the File I/O palette) reads data from the logfile. Note that this function has outputs containing the group name and the channel(s) name(s).

• The data that are read from the file are converted to a ordnary 2-dimensional array containing the three columns of data (t, u, y). The convertion is made by the Convert from Dynamic Data function which is on the Express / Signal Manipulation palette. This conversion function can be configured (i.e. selecting the correct data type) by double-clicking it, cf. the dialog window shown in the figure below.

• The three Index Array functions are used to extract each of the columns from the data and create 1 dimensional arrays, labeled Array t (1D), Array u (1D) and Array y (1D).

• The Block diagram also contains code for displaying the array in an array indicator and to display the size of the array on the Front panel.

• The TDMS Viewer function is a graph tool for quickly plotting the channels in the logfile.

Dialog window of the Convert from Dynamic Data function

LVM• LVM (LabVIEW Measurement) which are text files that

contain data that can be read by a human being because numbers are represented as text. For example, the number 0.231 is stored as the text (or string) "0.231". A large benfit of storing numerical data in the text format is that the file can be opened and displayed in any tool that supports text files, e.g. MS Word, Notepad, Excel, Matlab, Web browsers. (It may be necessary to change the file name extension from lvm to e.g. txt or dat before opening the file in such tools). Thus, text files provides great portability. However, the text files will be larger than if the data were written in a binary format, and the data are stored with less accuracy, but these issues are not important on ordinary PC's, but may be important on dedicated computers with limited storage.

LVM

• As a general rule I suggest that you use the binary file format (TDMS) unless you really need the text format (LVM).

• Both TDMS (binary) and LVM (text) files writing and reading are described in the subsequent sections.

Writing data to a LVM file

Writing data to a LVM file• In this example the data are writting to a file with the Write

To Measurement File function, which is an Express VI. (The Write to Spreadsheet File function could also have been used.)

• The boolean (switch) Start_stop_file_writing is used to start (enable) and stop the continuous file writing. This booleanterminal (control) is connected to the Enable input to the Write To Measurement File function.

• Two signals, u and y, are collected with a Merge Signalsfunction (which is on the Express / Signal Manipulationpalette) before they are fed into the the Write To Measurement File function. Regarding writing the time signal, cf. the comments about the X Value Columns in the list of comments below.

Writing data to a LVM file• By double-clicking the Write To Measurement File function in the block

diagram a dialog window opens, see the figure below.

Writing data to a LVM file• File Name box: Here you can select the file name, but if you have wired a file name control (or

constant) to the File Name input to this Express VI, the wired name is used instead (this is the case in the example above).

• File Format radio buttons: Here you select the data format: LVM or TDMS. (The TDM file format also available is a simpler and not so flexible data format. TDMS is a newer format.) It seems it is not possible to change the LVM file name extension into e.g. TXT (but you can of course do this change manually using Windows Explorer).

• Segment Headers: By selecting No headers the files will contain just plain data, which is convenient if you plan to import your data into tools as Excel, Matlab, etc. If you choose to have header(s), you will have to remove the header (perhaps manually) from the data file before you can load the file into tools mentioned above.

• X Value Columns: The X values are actually the time stamps. By selecting One column only the file will contain one left column containing the time stamps. These time stamps are recorded and written to the file automatically, so you do not have to create any time signal yourself.

• Delimiter: The delimiter is the character that separates columns. • File Description: The text you write here will appear in the header of the file, unless you have

selected the No headers options mentioned above. • Note: There is no way to define the group name or the channels names for the data in the Write

to Measurement File Express VI. • You may wonder why I did not use the Write To Measurement File function for writing the data in

TDMS format (cf. the previous section). In stead I used the TDMS Write function. The reason is that the channel group name and the channel names can not be defined in the Write To Measurement File function, but if this is not an issue, you may of course use the Write To Measurement File function for writing TDMS data.

Writing data to a LVM file• In our example, the result of running previous VI is a text file named

logfile1.lvm. The figure below shows the file opened in Notepad from one experiment. (The cycle time of the While loop was set to 0.1 s, cf.the block diagram shown above. As you see, the actual cycle time varies a little, but it is very close to the specified cycle time of 0.1 s.)

logfile1.lvm file opened in Notepad

Reading data from a LVM file

• The VI does not contain any While loop since the read from file operation is done once. (This is the typical situation.)

• The Read From Measurement File function reads data from the logfile. (By double-clicking the function a dialog window is opened. The parameters in this window should be self-explanatory.

• The data that are read from the file are converted to a ordnary 2-dimensional array containing the three columns of data, cf. the logfile1.lvm shown above. The conversion is made by the Convert from Dynamic Data function which is on the Express / Signal Manipulation palette. Note that you can configure the Convert from Dynamic Data function (i.e. selecting the correct data type) by double-clicking it.

• The three Index Array functions are used to extract each of the columns from the data and create 1 dimensional arrays, labeled Array t (1D), Array u (1D) and Array y (1D).

• The Signal indicator is a Graph indicator which is created by right-clicking on the signal line out from the Expresss VI. The x-axis shows the time-index (integer). You can configure the Graph via its Property dialog window so that the x-axis shows time in secons (or minutes etc.).

• The Block diagram also contains code for displaying the array in an array indicator and to display the size of the array on the Front panel.